Stowage/clamav
2
0
Fork 0
mirror of https://github.com/Cisco-Talos/clamav.git synced 2025-10-19 10:23:17 +00:00
Code Issues Projects Releases Packages Wiki Activity
clamav/libclamav/scanners.c

6716 lines
224 KiB
C
Raw Normal View History

Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
/*
Bump copyright dates for 2025
2025-02-14 10:24:30 -05:00
* Copyright (C) 2013-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
XZ file type support
2013-10-08 17:17:44 -04:00
* Copyright (C) 2007-2013 Sourcefire, Inc.
mail: scan text attachments and decoded base64 bodies also with type 4 sigs (bb#378) git-svn: trunk@3615
2008-02-11 18:34:28 +00:00
*
update copyrights and stick more files to GPLv2; move and add more credits to the AUTHORS file; add COPYING.BSD git-svn: trunk@3749
2008-04-02 15:24:51 +00:00
* Authors: Tomasz Kojm
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
*
* This program is free software; you can redistribute it and/or modify
update some copyrights and stick to GPL v2 git-svn: trunk@3003
2007-03-31 20:31:04 +00:00
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
update GPL headers with new address for FSF git-svn: trunk@1901
2006-04-09 19:59:28 +00:00
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
*/
Support for clamav-config.h git-svn: trunk@246
2004-02-06 13:46:08 +00:00
#if HAVE_CONFIG_H
#include "clamav-config.h"
#endif
fmap, scanners.c: win32 fixes
2009-10-12 23:32:27 +02:00
#ifndef _WIN32
mbox to fmap
2009-09-10 03:19:43 +02:00
#include <sys/time.h>
fmap, scanners.c: win32 fixes
2009-10-12 23:32:27 +02:00
#endif
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
#include <libgen.h>
add missing check for file open failure git-svn: trunk@3904
2008-07-03 13:03:26 +00:00
#include <errno.h>
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#include <sys/types.h>
#include <sys/stat.h>
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
#include <stdbool.h>
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
#ifdef HAVE_UNISTD_H
add support for old fashioned tar archives git-svn: trunk@1421
2005-03-22 21:26:27 +00:00
#include <unistd.h>
apply w32 patches from NJH git-svn: trunk@2359
2006-10-09 15:23:50 +00:00
#endif
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
#ifdef HAVE_SYS_PARAM_H
fix compilation error git-svn: trunk@1093
2004-11-13 14:47:20 +00:00
#include <sys/param.h>
apply w32 patches from NJH git-svn: trunk@2359
2006-10-09 15:23:50 +00:00
#endif
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#include <fcntl.h>
#include <dirent.h>
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
#ifdef HAVE_SYS_TIMES_H
#include <sys/times.h>
#endif
new method of file type detection; HTML normalisation git-svn: trunk@648
2004-07-02 23:00:58 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
#define DCONF_ARCH ctx->dconf->archive
#define DCONF_DOC ctx->dconf->doc
#define DCONF_MAIL ctx->dconf->mail
dynamic configuration support git-svn: trunk@2603
2007-01-09 20:06:51 +00:00
#define DCONF_OTHER ctx->dconf->other
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
#include <zlib.h>
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
#include "clamav_rust.h"
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#include "clamav.h"
#include "others.h"
dynamic configuration support git-svn: trunk@2603
2007-01-09 20:06:51 +00:00
#include "dconf.h"
libclamav: pe: integrate Petite unpacker from aCaB (not yet activated) git-svn: trunk@715
2004-08-02 17:09:06 +00:00
#include "scanners.h"
Use new patter matching algorithm. Cleanup. git-svn: trunk@674
2004-07-19 17:54:40 +00:00
#include "matcher-ac.h"
#include "matcher-bm.h"
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#include "matcher.h"
OLE2 git-svn: trunk@200
2004-01-23 11:17:16 +00:00
#include "ole2_extract.h"
#include "vba_extract.h"
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
#include "xlm_extract.h"
support files compressed with compress.exe git-svn: trunk@535
2004-05-02 00:51:01 +00:00
#include "msexpand.h"
minor code cleanup git-svn: trunk@1981
2006-05-18 11:29:24 +00:00
#include "mbox.h"
bb11062 - adding support for external libmspack Conflicts: configure configure.ac
2016-03-24 12:26:04 -04:00
#include "libmspack.h"
activate pe and upx code git-svn: trunk@651
2004-07-05 23:50:55 +00:00
#include "pe.h"
minor code cleanup git-svn: trunk@1981
2006-05-18 11:29:24 +00:00
#include "elf.h"
new method of file type detection; HTML normalisation git-svn: trunk@648
2004-07-02 23:00:58 +00:00
#include "filetypes.h"
#include "htmlnorm.h"
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
#include "untar.h"
eliminate false positive matches of Exploit.JPEG.Comment git-svn: trunk@949
2004-09-30 00:26:52 +00:00
#include "special.h"
minor cleanup git-svn: trunk@1282
2005-01-26 17:45:25 +00:00
#include "binhex.h"
minor code cleanup git-svn: trunk@1981
2006-05-18 11:29:24 +00:00
/* #include "uuencode.h" */
fix other implicit function declarations git-svn: trunk@1903
2006-04-10 09:59:51 +00:00
#include "tnef.h"
inital support for SIS files git-svn: trunk@1785
2005-12-15 02:02:29 +00:00
#include "sis.h"
Send ctx to pdf git-svn: trunk@1851
2006-03-08 15:37:52 +00:00
#include "pdf.h"
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
#include "str.h"
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
#include "entconv.h"
commit experimental code from Edvin for extracting embedded objects from RTF files git-svn: trunk@2534
2006-12-04 00:10:46 +00:00
#include "rtf.h"
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
#include "unarj.h"
fmap, scanners.c: win32 fixes
2009-10-12 23:32:27 +02:00
#include "nsis/nulsft.h"
preliminary autoit support (ea05/ea06) minor fixes coming up later git-svn: trunk@3340
2007-10-30 18:53:25 +00:00
#include "autoit.h"
support for generic text normalizer (CL_TYPE_SCRIPT) git-svn: trunk@3584
2008-02-04 21:38:34 +00:00
#include "textnorm.h"
new zip code git-svn: trunk@1987
2006-05-22 16:44:11 +00:00
#include "unzip.h"
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
#include "dlp.h"
fix type 7 signature matching (bb #1368). git-svn: trunk@4788
2009-02-14 15:35:26 +00:00
#include "default.h"
add missing #include
2009-07-06 16:35:55 +02:00
#include "cpio.h"
missing stuff (-a !@(*&!(@*&)
2009-07-08 15:05:22 +02:00
#include "macho.h"
ishield-msi complete
2009-07-13 02:37:16 +02:00
#include "ishield.h"
UPgrade lzma SDK to version 9.20 Also fmapify
2011-06-18 01:44:38 +02:00
#include "7z_iface.h"
fmap in magicscan
2009-08-30 19:14:49 +02:00
#include "fmap.h"
cache cache over 256 trees
2009-08-11 12:23:14 +02:00
#include "cache.h"
cosmetics (bb#2207)
2011-03-11 20:30:45 +01:00
#include "events.h"
do more checks
2011-04-06 15:53:28 +02:00
#include "swf.h"
#include "jpeg.h"
Making the GIF parser more tolerant and supporting GIF overlays
2017-03-04 00:08:03 +01:00
#include "gif.h"
libclamav: add PNG validator based on pngcheck
2011-04-07 18:19:19 +02:00
#include "png.h"
Add support for scanning different types of iso9660 image files. The allowed sector size is within 2048 to 2448 (2352 raw + 96 sub). Right now only the only file system supported is plain iso9660 with optional Joliet extensions. Additionally files with multi extents and interleaved files are not supported. Finally, due to the multiple possible ways to interpret the content of a cd/dvd, I cannot guarantee that we scan the "right" files.
2011-11-14 21:23:15 +01:00
#include "iso9660.h"
Add support for UDF files Add support for specifically for Beginning Extended Area Descriptor (BEA01) type of UDF files.
2023-06-02 10:45:38 -07:00
#include "udf.h"
libclamav: stub work for DMG (bb #1570) and XAR (bb #3801) formats
2013-08-27 13:44:11 -04:00
#include "dmg.h"
#include "xar.h"
libclamav: include hfsplus from scanners.c
2013-09-26 11:12:54 -04:00
#include "hfsplus.h"
XZ file type support
2013-10-08 17:17:44 -04:00
#include "xz_iface.h"
fixed include warnings for windows build
2014-02-07 12:22:44 -05:00
#include "mbr.h"
libclamav: added GPT parsing for raw DMGs
2014-01-24 14:24:56 -05:00
#include "gpt.h"
libclamav: linked MBR and APM parsing filetypes: handled file misclassification as mbr
2014-02-06 19:01:26 -05:00
#include "apm.h"
Silence compiler warnings.
2014-07-09 13:16:31 -04:00
#include "ooxml.h"
Add proof-of-concept XDP support. This feature requires libxml2 support. This commit bumps FLEVEL and introduces a new filetype based on the expected XML namespace for XDP files.
2014-07-31 19:11:22 -04:00
#include "xdp.h"
json_api: returns error codes or CL_SUCCESS w or w/o JSON doc/ppt/xls: propagates error codes to topmost lvl w/ premature termination
2014-04-24 14:22:00 -04:00
#include "json_api.h"
added basic framework for msxml processing
2015-03-10 15:33:32 -04:00
#include "msxml.h"
tiff: added heuristic check for invalid tiff IFD fields
2015-07-30 17:37:05 -04:00
#include "tiff.h"
various name changes and dereference fix
2015-12-08 17:28:49 -05:00
#include "hwp.h"
adding a few minor changes to eliminate warnings, the most interesting of which was a missing function prototype for cache_get_MD5.
2017-08-10 15:40:52 -04:00
#include "msdoc.h"
PE parsing code improvements, db loading bug fixes Consolidate the PE parsing code into one function. I tried to preserve all existing functionality from the previous, distinct implementations to a large extent (with the exceptions mentioned below). If I noticed potential bugs/improvements, I added a TODO statement about those so that they can be fixed in a smaller commit later. Also, there are more TODOs in places where I'm not entirely sure why certain actions are performed - more research is needed for these. I'm submitting a pull request now so that regression testing can be done, and because merging what I have thus far now will likely have fewer conflicts than if I try to merge later PE parsing code improvements: - PEs without all 16 data directories are parsed more appropriately now - Added lots more debug statements Also: - Allow MAX_BC and MAX_TRACKED_PCRE to be specified via CFLAGS When doing performance testing with the latest CVD, MAX_BC and MAX_TRACKED_PCRE need to be raised to track all the events. Allow these to be specified via CFLAGS by not redefining them if they are already defined - Fix an issue preventing wildcard sizes in .MDB/.MSB rules I'm not sure what the original intent of the check I removed was, but it prevents using wildcard sizes in .MDB/.MSB rules. AFAICT these wildcard sizes should be handled appropriately by the MD5 section hash computation code, so I don't think a check on that is needed. - Fix several issues related to db loading - .imp files will now get loaded if they exist in a directory passed via clamscan's '-d' flag - .pwdb files will now get loaded if they exist in a directory passed via clamscan's '-d' flag even when compiling without yara support - Changes to .imp, .ign, and .ign2 files will now be reflected in calls to cl_statinidir and cl_statchkdir (and also .pwdb files, even when compiling without yara support) - The contents of .sfp files won't be included in some of the signature counts, and the contents of .cud files will be - Any local.gdb files will no longer be loaded twice - For .imp files, you are no longer required to specify a minimum flevel for wildcard rules, since this isn't needed
2019-01-08 00:09:08 -05:00
#include "execs.h"
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
#include "egg.h"
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
// libclamunrar_iface
#include "unrar_iface.h"
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#include <bzlib.h>
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
#include <fcntl.h>
Use OpenSSL for hashing.
2014-02-08 00:31:12 -05:00
#include <string.h>
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
cl_error_t cli_magic_scan_dir(const char *dir, cli_ctx *ctx, uint32_t attributes)
mail: scan text attachments and decoded base64 bodies also with type 4 sigs (bb#378) git-svn: trunk@3615
2008-02-11 18:34:28 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
DIR *dd = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct dirent *dent;
STATBUF statbuf;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
char *fname = NULL;
mail: scan text attachments and decoded base64 bodies also with type 4 sigs (bb#378) git-svn: trunk@3615
2008-02-11 18:34:28 +00:00
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if ((dd = opendir(dir)) != NULL) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while ((dent = readdir(dd))) {
if (dent->d_ino) {
if (strcmp(dent->d_name, ".") && strcmp(dent->d_name, "..")) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* build the full name */
Rename clamav memory allocation functions We have some special functions to wrap malloc, calloc, and realloc to make sure we don't allocate more than some limit, similar to the max-filesize and max-scansize limits. Our wrappers are really only needed when allocating memory for scans based on untrusted user input, where a scan file could have bytes that claim you need to allocate some ridiculous amount of memory. Right now they're named: - cli_malloc - cli_calloc - cli_realloc - cli_realloc2 ... and these names do not convey their purpose This commit renames them to: - cli_max_malloc - cli_max_calloc - cli_max_realloc - cli_max_realloc2 The realloc ones also have an additional feature in that they will not free your pointer if you try to realloc to 0 bytes. Freeing the memory is undefined by the C spec, and only done with some realloc implementations, so this stabilizes on the behavior of not doing that, which should prevent accidental double-free's. So for the case where you may want to realloc and do not need to have a maximum, this commit adds the following functions: - cli_safer_realloc - cli_safer_realloc2 These are used for the MPOOL_REALLOC and MPOOL_REALLOC2 macros when MPOOL is disabled (e.g. because mmap-support is not found), so as to match the behavior in the mpool_realloc/2 functions that do not make use of the allocation-limit.
2022-05-09 14:28:34 -07:00
fname = malloc(strlen(dir) + strlen(dent->d_name) + 2);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!fname) {
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cli_dbgmsg("cli_magic_scan_dir: Unable to allocate memory for filename\n");
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EMEM;
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
sprintf(fname, "%s" PATHSEP "%s", dir, dent->d_name);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* stat the file */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (LSTAT(fname, &statbuf) != -1) {
if (S_ISDIR(statbuf.st_mode) && !S_ISLNK(statbuf.st_mode)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = cli_magic_scan_dir(fname, ctx, attributes);
if (CL_SUCCESS != status) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
if (S_ISREG(statbuf.st_mode)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = cli_magic_scan_file(fname, ctx, dent->d_name, attributes);
if (CL_SUCCESS != status) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
}
}
free(fname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fname = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("cli_magic_scan_dir: Can't open directory %s.\n", dir);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EOPEN;
goto done;
}
done:
if (NULL != dd) {
closedir(dd);
}
if (NULL != fname) {
free(fname);
mail: scan text attachments and decoded base64 bodies also with type 4 sigs (bb#378) git-svn: trunk@3615
2008-02-11 18:34:28 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
return status;
mail: scan text attachments and decoded base64 bodies also with type 4 sigs (bb#378) git-svn: trunk@3615
2008-02-11 18:34:28 +00:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/**
* @brief Scan the metadata using cli_matchmeta()
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
*
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
* @param metadata unrar metadata structure
* @param ctx scanning context structure
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
* @param files
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
* @return cl_error_t Returns CL_SUCCESS if nothing found, CL_VIRUS if something found, CL_EUNPACK if encrypted.
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
*/
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
static cl_error_t cli_unrar_scanmetadata(unrar_metadata_t *metadata, cli_ctx *ctx, unsigned int files)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
initial rewrite of limits git-svn-id: file:///var/lib/svn/clamav-devel/branches/newlimits@3591 77e5149b-7576-45b1-b177-96237e5ba77b
2008-02-06 21:19:10 +00:00
cli_dbgmsg("RAR: %s, crc32: 0x%x, encrypted: %u, compressed: %u, normal: %u, method: %u, ratio: %u\n",
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
metadata->filename, metadata->crc, metadata->encrypted, (unsigned int)metadata->pack_size,
(unsigned int)metadata->unpack_size, metadata->method,
metadata->pack_size ? (unsigned int)(metadata->unpack_size / metadata->pack_size) : 0);
scan meta-data in RAR files git-svn: trunk@1400
2005-03-18 01:27:45 +00:00
Add LHA/LZH archive support File type magic signatures chosen based on the extensions supported by Rust delharc crate. See: https://docs.rs/delharc/latest/delharc/
2023-11-07 23:00:30 -05:00
if (CL_VIRUS == cli_matchmeta(ctx, metadata->filename, metadata->pack_size, metadata->unpack_size, metadata->encrypted, files, metadata->crc)) {
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
status = CL_VIRUS;
} else if (SCAN_HEURISTIC_ENCRYPTED_ARCHIVE && metadata->encrypted) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("RAR: Encrypted files found in archive.\n");
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
status = CL_EUNPACK;
extract and scan RAR archives file-by-file (bb#141) git-svn: trunk@2615
2007-01-13 00:01:39 +00:00
}
scan meta-data in RAR files git-svn: trunk@1400
2005-03-18 01:27:45 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
return status;
improve handling of RAR archives git-svn: trunk@2648
2007-01-28 20:22:16 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
static cl_error_t cli_scanrar_file(const char *filepath, int desc, cli_ctx *ctx)
extract and scan RAR archives file-by-file (bb#141) git-svn: trunk@2615
2007-01-13 00:01:39 +00:00
{
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
cl_error_t status = CL_EPARSE;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
cl_unrar_error_t unrar_ret = UNRAR_ERR;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
unsigned int file_count = 0;
extract and scan RAR archives file-by-file (bb#141) git-svn: trunk@2615
2007-01-13 00:01:39 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
uint32_t nEncryptedFilesFound = 0;
extract and scan RAR archives file-by-file (bb#141) git-svn: trunk@2615
2007-01-13 00:01:39 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
void *hArchive = NULL;
use B-M to handle .hdb and .fp databases and other minor changes git-svn: trunk@3425
2007-12-15 20:34:31 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
char *comment = NULL;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
uint32_t comment_size = 0;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
unrar_metadata_t metadata;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
char *filename_base = NULL;
char *extract_fullpath = NULL;
char *comment_fullpath = NULL;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
UNUSEDPARAM(desc);
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (filepath == NULL || ctx == NULL) {
cli_dbgmsg("RAR: Invalid arguments!\n");
return CL_EARG;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
cli_dbgmsg("in scanrar()\n");
/* Zero out the metadata struct before we read the header */
memset(&metadata, 0, sizeof(unrar_metadata_t));
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/*
* Open the archive.
*/
if (UNRAR_OK != (unrar_ret = cli_unrar_open(filepath, &hArchive, &comment, &comment_size, cli_debug_flag))) {
if (unrar_ret == UNRAR_ENCRYPTED) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("RAR: Encrypted main header\n");
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = CL_SUCCESS;
nEncryptedFilesFound += 1;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (unrar_ret == UNRAR_EMEM) {
status = CL_EMEM;
goto done;
Fix file access issue if in low privelege process Removing problematic call to convert file descriptors to filepaths. Added filename and tempfile names to scandesc calls in clamd. Added a general scan option to treat the scan engine as unprivileged, meaning that the scan engine will not have read access to the file. Added check to drop a temp file for RAR's where the we don't have read access to the filepath provided (i.e. unprivileged is set, or access() check fails).
2020-01-23 17:42:33 -08:00
} else if (unrar_ret == UNRAR_EOPEN) {
status = CL_EOPEN;
goto done;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
} else {
status = CL_EFORMAT;
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/* If the archive header had a comment, write it to the comment dir. */
if ((comment != NULL) && (comment_size > 0)) {
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
if (ctx->engine->keeptmp) {
int comment_fd = -1;
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(comment_fullpath = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "comments"))) {
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
status = CL_EMEM;
goto done;
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
comment_fd = open(comment_fullpath, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, 0600);
if (comment_fd < 0) {
cli_dbgmsg("RAR: ERROR: Failed to open output file\n");
} else {
cli_dbgmsg("RAR: Writing the archive comment to temp file: %s\n", comment_fullpath);
if (0 == write(comment_fd, comment, comment_size)) {
cli_dbgmsg("RAR: ERROR: Failed to write to output file\n");
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
}
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
close(comment_fd);
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
}
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
}
/* Scan the comment */
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_magic_scan_buff(comment, comment_size, ctx, NULL, LAYER_ATTRIBUTES_NONE);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status != CL_SUCCESS) {
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/*
* Read & scan each file header.
* Extract & scan each file.
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
*
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
* Skip files if they will exceed max filesize or max scansize.
* Count the number of encrypted file headers and encrypted files.
* - Alert if there are encrypted files,
* if the Heuristic for encrypted archives is enabled,
* and if we have not detected a signature match.
*/
do {
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/* Zero out the metadata struct before we read the header */
memset(&metadata, 0, sizeof(unrar_metadata_t));
/*
* Get the header information for the next file in the archive.
*/
unrar_ret = cli_unrar_peek_file_header(hArchive, &metadata);
if (unrar_ret != UNRAR_OK) {
if (unrar_ret == UNRAR_ENCRYPTED) {
/* Found an encrypted file header, must skip. */
cli_dbgmsg("RAR: Encrypted file header, unable to reading file metadata and file contents. Skipping file...\n");
nEncryptedFilesFound += 1;
if (UNRAR_OK != cli_unrar_skip_file(hArchive)) {
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("RAR: Failed to skip file. RAR archive extraction has failed.\n");
break;
}
} else if (unrar_ret == UNRAR_BREAK) {
/* No more files. Break extraction loop. */
cli_dbgmsg("RAR: No more files in archive.\n");
break;
} else {
/* Memory error or some other error reading the header info. */
cli_dbgmsg("RAR: Error (%u) reading file header!\n", unrar_ret);
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
file_count += 1;
/*
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Scan the metadata for the file in question since the content was clean, or we're running in all-match.
*/
Removing unRAR SFX Check from scanners.c. Flawed feature was skipping scans of files in RAR archives that had the same CRC in the RAR file entry header as a previously scanned entry. Archives CRC's cannot be trusted. Removing the SFX Check eliminated false negatives in regression testing.
2018-09-24 15:01:22 -04:00
status = cli_unrar_scanmetadata(&metadata, ctx, file_count);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status == CL_EUNPACK) {
nEncryptedFilesFound += 1;
} else if (status != CL_SUCCESS) {
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
break;
}
/* Check if we've already exceeded the scan limit */
if (cli_checklimits("RAR", ctx, 0, 0, 0))
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (metadata.is_dir) {
/* Entry is a directory. Skip. */
cli_dbgmsg("RAR: Found directory. Skipping to next file.\n");
if (UNRAR_OK != cli_unrar_skip_file(hArchive)) {
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("RAR: Failed to skip directory. RAR archive extraction has failed.\n");
break;
}
} else if (cli_checklimits("RAR", ctx, metadata.unpack_size, 0, 0)) {
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
/* File size exceeds maxfilesize, must skip extraction.
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Although we may be able to scan the metadata */
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
cli_dbgmsg("RAR: Next file is too large (%" PRIu64 " bytes); it would exceed max scansize. Skipping to next file.\n", metadata.unpack_size);
if (UNRAR_OK != cli_unrar_skip_file(hArchive)) {
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("RAR: Failed to skip file. RAR archive extraction has failed.\n");
break;
}
} else if (metadata.encrypted != 0) {
/* Found an encrypted file, must skip. */
cli_dbgmsg("RAR: Encrypted file, unable to extract file contents. Skipping file...\n");
nEncryptedFilesFound += 1;
if (UNRAR_OK != cli_unrar_skip_file(hArchive)) {
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("RAR: Failed to skip file. RAR archive extraction has failed.\n");
break;
}
} else {
/*
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
* Extract the file...
*/
Addressed CID 432800
2024-03-28 13:36:49 -07:00
if (0 != metadata.filename[0]) {
Windows: Fix filepath basename issue On Windows, the cli_basename function should treat both '/' and '\' as path separators. Most Windows APIs also accept both. On Linux/Unix, it makes sense when using a filepath that is more for informational purposes or where it may have come from a Windows system, to treat the '\' as a path separator. But in situations where the the path is needed for some critical action, like moving or deleting a file, we can't treat it as a path separator.
2025-08-11 18:02:09 -04:00
(void)cli_basename(metadata.filename, strlen(metadata.filename), &filename_base, true /* posix_support_backslash_pathsep */);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
if (!(ctx->engine->keeptmp) ||
(NULL == filename_base)) {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
extract_fullpath = cli_gentemp(ctx->this_layer_tmpdir);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
} else {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
extract_fullpath = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, filename_base);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (NULL == extract_fullpath) {
cli_dbgmsg("RAR: Memory error allocating filename for extracted file.");
status = CL_EMEM;
break;
}
cli_dbgmsg("RAR: Extracting file: %s to %s\n", metadata.filename, extract_fullpath);
unrar_ret = cli_unrar_extract_file(hArchive, extract_fullpath, NULL);
if (unrar_ret != UNRAR_OK) {
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
/*
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
* Some other error extracting the file
*/
cli_dbgmsg("RAR: Error extracting file: %s\n", metadata.filename);
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
/* TODO:
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
* may need to manually skip the file depending on what, specifically, cli_unrar_extract_file() returned.
*/
} else {
bb12285 - Change to set read permissions on files extracted from RAR archive in case permissions were inadequate.
2019-05-28 14:40:40 -07:00
/*
* File should be extracted...
* ... make sure we have read permissions to the file.
*/
#ifdef _WIN32
if (0 != _access_s(extract_fullpath, R_OK)) {
#else
if (0 != access(extract_fullpath, R_OK)) {
#endif
cli_dbgmsg("RAR: Don't have read permissions, attempting to change file permissions to make it readable..\n");
#ifdef _WIN32
if (0 != _chmod(extract_fullpath, _S_IREAD)) {
#else
if (0 != chmod(extract_fullpath, S_IRUSR | S_IRGRP)) {
#endif
cli_dbgmsg("RAR: Failed to change permission bits so the extracted file is readable..\n");
}
}
/*
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
* ... scan the extracted file.
*/
cli_dbgmsg("RAR: Extraction complete. Scanning now...\n");
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_magic_scan_file(extract_fullpath, ctx, filename_base, LAYER_ATTRIBUTES_NONE);
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (status == CL_EOPEN) {
cli_dbgmsg("RAR: File not found, Extraction failed!\n");
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Don't abort the scan just because one file failed to extract.
status = CL_SUCCESS;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
} else {
/* Delete the tempfile if not --leave-temps */
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(extract_fullpath)) {
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
cli_dbgmsg("RAR: Failed to unlink the extracted file: %s\n", extract_fullpath);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status != CL_SUCCESS) {
// Bail out if "virus" and also if exceeded scan maximums, etc.
goto done;
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
}
}
}
/* Free up that the filepath */
if (NULL != extract_fullpath) {
free(extract_fullpath);
extract_fullpath = NULL;
}
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/*
Fix fmap NULL deref in preclass bytecode hook If using a bytecode signature that makes use of the BC_PRECLASS hook and if it alerts, a NULL dereference may occur. This change fixes that. Also fixed unrelated memory leaks introduced recently when adding file name extraction to the zip parser and rar parser.
2020-04-14 11:52:12 -04:00
* Free up any malloced metadata...
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
*/
Fix fmap NULL deref in preclass bytecode hook If using a bytecode signature that makes use of the BC_PRECLASS hook and if it alerts, a NULL dereference may occur. This change fixes that. Also fixed unrelated memory leaks introduced recently when adding file name extraction to the zip parser and rar parser.
2020-04-14 11:52:12 -04:00
if (NULL != filename_base) {
free(filename_base);
filename_base = NULL;
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
} while (status == CL_SUCCESS);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status == CL_BREAK) {
status = CL_SUCCESS;
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
done:
if (NULL != comment) {
free(comment);
comment = NULL;
}
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (NULL != comment_fullpath) {
if (!ctx->engine->keeptmp) {
cli_rmdirs(comment_fullpath);
}
free(comment_fullpath);
comment_fullpath = NULL;
}
cli_unrar_close() was being called too early git-svn: trunk@2636
2007-01-20 11:38:54 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (NULL != hArchive) {
cli_unrar_close(hArchive);
hArchive = NULL;
}
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (NULL != filename_base) {
free(filename_base);
filename_base = NULL;
}
extract and scan RAR archives file-by-file (bb#141) git-svn: trunk@2615
2007-01-13 00:01:39 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (NULL != extract_fullpath) {
free(extract_fullpath);
extract_fullpath = NULL;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if ((CL_VIRUS != status) && (nEncryptedFilesFound > 0)) {
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
/* If user requests enabled the Heuristic for encrypted archives... */
if (SCAN_HEURISTIC_ENCRYPTED_ARCHIVE) {
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
if (CL_VIRUS == cli_append_potentially_unwanted(ctx, "Heuristics.Encrypted.RAR")) {
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
status = CL_VIRUS;
}
}
}
cli_dbgmsg("RAR: Exit code: %d\n", status);
fixed RAR support git-svn: trunk@404
2004-03-16 19:39:49 +00:00
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
return status;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
static cl_error_t cli_scanrar(cli_ctx *ctx)
{
cl_error_t status = CL_SUCCESS;
const char *filepath = NULL;
int fd = -1;
char *tmpname = NULL;
int tmpfd = -1;
#ifdef _WIN32
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if ((SCAN_UNPRIVILEGED) || (NULL == ctx->fmap->path) || (0 != _access_s(ctx->fmap->path, R_OK))) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
#else
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if ((SCAN_UNPRIVILEGED) || (NULL == ctx->fmap->path) || (0 != access(ctx->fmap->path, R_OK))) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
#endif
/* If map is not file-backed have to dump to file for scanrar. */
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
status = fmap_dump_to_file(ctx->fmap, ctx->fmap->path, ctx->this_layer_tmpdir, &tmpname, &tmpfd, 0, SIZE_MAX);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (status != CL_SUCCESS) {
cli_dbgmsg("cli_magic_scan: failed to generate temporary file.\n");
goto done;
}
filepath = tmpname;
fd = tmpfd;
} else {
/* Use the original file and file descriptor. */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
filepath = ctx->fmap->path;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fd = fmap_fd(ctx->fmap);
}
/* scan file */
status = cli_scanrar_file(filepath, fd, ctx);
if ((NULL == tmpname) && (CL_EOPEN == status)) {
/*
* Failed to open the file using the original filename.
* Try writing the file descriptor to a temp file and try again.
*/
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
status = fmap_dump_to_file(ctx->fmap, ctx->fmap->path, ctx->this_layer_tmpdir, &tmpname, &tmpfd, 0, SIZE_MAX);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (status != CL_SUCCESS) {
cli_dbgmsg("cli_magic_scan: failed to generate temporary file.\n");
goto done;
}
filepath = tmpname;
fd = tmpfd;
/* try to scan again */
status = cli_scanrar_file(filepath, fd, ctx);
}
done:
if (tmpfd != -1) {
/* If dumped tempfile, need to cleanup */
close(tmpfd);
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
status = CL_EUNLINK;
}
}
}
if (tmpname != NULL) {
free(tmpname);
}
return status;
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/**
* @brief Scan the metadata using cli_matchmeta()
*
* @param metadata egg metadata structure
* @param ctx scanning context structure
* @param files number of files
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
* @return cl_error_t Returns CL_SUCCESS if nothing found, CL_VIRUS if something found, CL_EUNPACK if encrypted.
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
*/
static cl_error_t cli_egg_scanmetadata(cl_egg_metadata *metadata, cli_ctx *ctx, unsigned int files)
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
cli_dbgmsg("EGG: %s, encrypted: %u, compressed: %u, normal: %u, ratio: %u\n",
metadata->filename, metadata->encrypted, (unsigned int)metadata->pack_size,
(unsigned int)metadata->unpack_size,
metadata->pack_size ? (unsigned int)(metadata->unpack_size / metadata->pack_size) : 0);
Add LHA/LZH archive support File type magic signatures chosen based on the extensions supported by Rust delharc crate. See: https://docs.rs/delharc/latest/delharc/
2023-11-07 23:00:30 -05:00
if (CL_VIRUS == cli_matchmeta(ctx, metadata->filename, metadata->pack_size, metadata->unpack_size, metadata->encrypted, files, 0)) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
status = CL_VIRUS;
} else if (SCAN_HEURISTIC_ENCRYPTED_ARCHIVE && metadata->encrypted) {
cli_dbgmsg("EGG: Encrypted files found in archive.\n");
status = CL_EUNPACK;
}
return status;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
static cl_error_t cli_scanegg(cli_ctx *ctx)
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
{
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cl_error_t status = CL_SUCCESS;
cl_error_t egg_ret;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
unsigned int file_count = 0;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
uint32_t nEncryptedFilesFound = 0;
void *hArchive = NULL;
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
char **comments = NULL;
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
uint32_t nComments = 0;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
cl_egg_metadata metadata;
char *filename_base = NULL;
char *extract_fullpath = NULL;
char *comment_fullpath = NULL;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
char *extract_filename = NULL;
char *extract_buffer = NULL;
size_t extract_buffer_len = 0;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
if (ctx == NULL) {
cli_dbgmsg("EGG: Invalid arguments!\n");
return CL_EARG;
}
cli_dbgmsg("in scanegg()\n");
/* Zero out the metadata struct before we read the header */
memset(&metadata, 0, sizeof(cl_egg_metadata));
/*
* Open the archive.
*/
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (CL_SUCCESS != (egg_ret = cli_egg_open(ctx->fmap, &hArchive, &comments, &nComments))) {
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (egg_ret == CL_EUNPACK) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
cli_dbgmsg("EGG: Encrypted main header\n");
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
nEncryptedFilesFound += 1;
status = CL_SUCCESS;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
goto done;
}
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (egg_ret == CL_EMEM) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
status = CL_EMEM;
goto done;
} else {
status = CL_EFORMAT;
goto done;
}
}
/* If the archive header had a comment, write it to the comment dir. */
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
if (comments != NULL) {
uint32_t i;
for (i = 0; i < nComments; i++) {
/*
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Drop the comment to a temp file, if requested
*/
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
if (ctx->engine->keeptmp) {
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
int comment_fd = -1;
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
size_t prefixLen = strlen("comments_") + 5;
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
char *prefix = (char *)malloc(prefixLen + 1);
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
snprintf(prefix, prefixLen, "comments_%u", i);
prefix[prefixLen] = '\0';
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(comment_fullpath = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, prefix))) {
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
free(prefix);
status = CL_EMEM;
goto done;
}
free(prefix);
comment_fd = open(comment_fullpath, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, 0600);
if (comment_fd < 0) {
cli_dbgmsg("EGG: ERROR: Failed to open output file\n");
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
} else {
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
cli_dbgmsg("EGG: Writing the archive comment to temp file: %s\n", comment_fullpath);
if (0 == write(comment_fd, comments[i], nComments)) {
cli_dbgmsg("EGG: ERROR: Failed to write to output file\n");
}
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
close(comment_fd);
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
}
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
free(comment_fullpath);
comment_fullpath = NULL;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
}
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
/*
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Scan the comment.
*/
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_magic_scan_buff(comments[i], strlen(comments[i]), ctx, NULL, LAYER_ATTRIBUTES_NONE);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status != CL_SUCCESS) {
Fixed Egg parser support for parsing archive comment headers and elimited potential infinite loop bug.
2019-05-24 10:00:35 -04:00
goto done;
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
}
}
/*
* Read & scan each file header.
* Extract & scan each file.
*
* Skip files if they will exceed max filesize or max scansize.
* Count the number of encrypted file headers and encrypted files.
* - Alert if there are encrypted files,
* if the Heuristic for encrypted archives is enabled,
* and if we have not detected a signature match.
*/
do {
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Zero out the metadata struct before we read the header */
Fix for cli_scanegg; changing memset to use sizeof() the correct struct.
2019-08-24 20:48:50 -04:00
memset(&metadata, 0, sizeof(cl_egg_metadata));
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/*
* Get the header information for the next file in the archive.
*/
egg_ret = cli_egg_peek_file_header(hArchive, &metadata);
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (egg_ret != CL_SUCCESS) {
if (egg_ret == CL_EUNPACK) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Found an encrypted file header, must skip. */
cli_dbgmsg("EGG: Encrypted file header, unable to reading file metadata and file contents. Skipping file...\n");
nEncryptedFilesFound += 1;
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (CL_SUCCESS != cli_egg_skip_file(hArchive)) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("EGG: Failed to skip file. EGG archive extraction has failed.\n");
break;
}
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
} else if (egg_ret == CL_BREAK) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* No more files. Break extraction loop. */
cli_dbgmsg("EGG: No more files in archive.\n");
break;
} else {
/* Memory error or some other error reading the header info. */
cli_dbgmsg("EGG: Error (%u) reading file header!\n", egg_ret);
break;
}
} else {
file_count += 1;
/*
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Scan the metadata for the file in question since the content was clean, or we're running in all-match.
*/
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
status = cli_egg_scanmetadata(&metadata, ctx, file_count);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status == CL_EUNPACK) {
nEncryptedFilesFound += 1;
} else if (status != CL_SUCCESS) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
break;
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Check if we've already exceeded the scan limit */
if (cli_checklimits("EGG", ctx, 0, 0, 0))
break;
if (metadata.is_dir) {
/* Entry is a directory. Skip. */
cli_dbgmsg("EGG: Found directory. Skipping to next file.\n");
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (CL_SUCCESS != cli_egg_skip_file(hArchive)) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("EGG: Failed to skip directory. EGG archive extraction has failed.\n");
break;
}
} else if (cli_checklimits("EGG", ctx, metadata.unpack_size, 0, 0)) {
/* File size exceeds maxfilesize, must skip extraction.
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Although we may be able to scan the metadata */
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
cli_dbgmsg("EGG: Next file is too large (%" PRIu64 " bytes); it would exceed max scansize. Skipping to next file.\n", metadata.unpack_size);
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (CL_SUCCESS != cli_egg_skip_file(hArchive)) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("EGG: Failed to skip file. EGG archive extraction has failed.\n");
break;
}
} else if (metadata.encrypted != 0) {
/* Found an encrypted file, must skip. */
cli_dbgmsg("EGG: Encrypted file, unable to extract file contents. Skipping file...\n");
nEncryptedFilesFound += 1;
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (CL_SUCCESS != cli_egg_skip_file(hArchive)) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/* Failed to skip! Break extraction loop. */
cli_dbgmsg("EGG: Failed to skip file. EGG archive extraction has failed.\n");
break;
}
} else {
/*
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
* Extract the file...
*/
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
cli_dbgmsg("EGG: Extracting file: %s\n", metadata.filename);
egg_ret = cli_egg_extract_file(hArchive, (const char **)&extract_filename, (const char **)&extract_buffer, &extract_buffer_len);
Converted egg parser to use cl_error_t instead of cl_egg_error_t.
2019-07-01 16:08:14 -04:00
if (egg_ret != CL_SUCCESS) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
/*
* Some other error extracting the file
*/
cli_dbgmsg("EGG: Error extracting file: %s\n", metadata.filename);
} else if (!extract_buffer || 0 == extract_buffer_len) {
/*
* Empty file. Skip.
*/
cli_dbgmsg("EGG: Skipping empty file: %s\n", metadata.filename);
Fixes filename memory leak for empty files in EGG archives.
2019-09-07 07:29:01 -07:00
if (NULL != extract_filename) {
free(extract_filename);
extract_filename = NULL;
}
if (NULL != extract_buffer) {
free(extract_buffer);
extract_buffer = NULL;
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
} else {
/*
* Drop to a temp file, if requested.
*/
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
if (NULL != metadata.filename) {
Windows: Fix filepath basename issue On Windows, the cli_basename function should treat both '/' and '\' as path separators. Most Windows APIs also accept both. On Linux/Unix, it makes sense when using a filepath that is more for informational purposes or where it may have come from a Windows system, to treat the '\' as a path separator. But in situations where the the path is needed for some critical action, like moving or deleting a file, we can't treat it as a path separator.
2025-08-11 18:02:09 -04:00
(void)cli_basename(metadata.filename, strlen(metadata.filename), &filename_base, true /* posix_support_backslash_pathsep */);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
if (ctx->engine->keeptmp) {
int extracted_fd = -1;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
if (NULL == filename_base) {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
extract_fullpath = cli_gentemp(ctx->this_layer_tmpdir);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
} else {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
extract_fullpath = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, filename_base);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
if (NULL == extract_fullpath) {
cli_dbgmsg("EGG: Memory error allocating filename for extracted file.");
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
status = CL_EMEM;
break;
}
extracted_fd = open(extract_fullpath, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, 0600);
if (extracted_fd < 0) {
cli_dbgmsg("EGG: ERROR: Failed to open output file\n");
} else {
cli_dbgmsg("EGG: Writing the extracted file contents to temp file: %s\n", extract_fullpath);
if (0 == write(extracted_fd, extract_buffer, extract_buffer_len)) {
cli_dbgmsg("EGG: ERROR: Failed to write to output file\n");
} else {
close(extracted_fd);
extracted_fd = -1;
}
}
}
/*
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
* Scan the extracted file...
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
*/
cli_dbgmsg("EGG: Extraction complete. Scanning now...\n");
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_magic_scan_buff(extract_buffer, extract_buffer_len, ctx, filename_base, LAYER_ATTRIBUTES_NONE);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status != CL_SUCCESS) {
goto done;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
if (NULL != filename_base) {
free(filename_base);
filename_base = NULL;
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
if (NULL != extract_filename) {
free(extract_filename);
extract_filename = NULL;
}
if (NULL != extract_buffer) {
free(extract_buffer);
extract_buffer = NULL;
}
}
/* Free up that the filepath */
if (NULL != extract_fullpath) {
free(extract_fullpath);
extract_fullpath = NULL;
}
}
}
if (ctx->engine->maxscansize && ctx->scansize >= ctx->engine->maxscansize) {
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
break;
}
/*
* TODO: Free up any malloced metadata...
*/
if (metadata.filename != NULL) {
free(metadata.filename);
metadata.filename = NULL;
}
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
} while (status == CL_SUCCESS);
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (status == CL_BREAK) {
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
done:
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (NULL != extract_filename) {
free(extract_filename);
extract_filename = NULL;
}
if (NULL != extract_buffer) {
free(extract_buffer);
extract_buffer = NULL;
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
if (NULL != comment_fullpath) {
free(comment_fullpath);
comment_fullpath = NULL;
}
if (NULL != hArchive) {
cli_egg_close(hArchive);
hArchive = NULL;
}
if (NULL != filename_base) {
free(filename_base);
filename_base = NULL;
}
if (metadata.filename != NULL) {
free(metadata.filename);
metadata.filename = NULL;
}
if (NULL != extract_fullpath) {
free(extract_fullpath);
extract_fullpath = NULL;
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if ((CL_VIRUS != status) && (nEncryptedFilesFound > 0)) {
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
/* If user requests enabled the Heuristic for encrypted archives... */
if (SCAN_HEURISTIC_ENCRYPTED_ARCHIVE) {
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
if (CL_VIRUS == cli_append_potentially_unwanted(ctx, "Heuristics.Encrypted.EGG")) {
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
status = CL_VIRUS;
}
}
}
cli_dbgmsg("EGG: Exit code: %d\n", status);
return status;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
static cl_error_t cli_scanarj(cli_ctx *ctx)
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
int file = 0;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
arj_metadata_t metadata;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
char *dir = NULL;
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
cli_dbgmsg("in cli_scanarj()\n");
bb#11514 - initialize ARJ metadata structures
2016-03-08 14:37:20 -05:00
memset(&metadata, 0, sizeof(arj_metadata_t));
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* generate the temporary directory */
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "arj-tmp")))
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return CL_EMEM;
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(dir, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("ARJ: Can't create temporary directory %s\n", dir);
free(dir);
return CL_ETMPDIR;
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
ret = cli_unarj_open(ctx->fmap, dir, &metadata);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ret != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
cli_rmdirs(dir);
free(dir);
cli_dbgmsg("ARJ: Error: %s\n", cl_strerror(ret));
return ret;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
do {
unarj leaks
2009-10-23 20:49:12 +02:00
metadata.filename = NULL;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = cli_unarj_prepare_file(dir, &metadata);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ret != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("ARJ: cli_unarj_prepare_file Error: %s\n", cl_strerror(ret));
break;
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
file++;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Add LHA/LZH archive support File type magic signatures chosen based on the extensions supported by Rust delharc crate. See: https://docs.rs/delharc/latest/delharc/
2023-11-07 23:00:30 -05:00
if (CL_VIRUS == cli_matchmeta(ctx, metadata.filename, metadata.comp_size, metadata.orig_size, metadata.encrypted, file, 0)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_rmdirs(dir);
free(dir);
return CL_VIRUS;
bb11580 - make cli_matchmeta() respect allmatch.
2016-06-08 16:25:34 -04:00
}
libclamav: cdb: drop FileType; cover ARJ, CAB, TAR, CPIO and 7Z
2010-01-14 23:32:35 +01:00
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if ((ret = cli_checklimits("ARJ", ctx, metadata.orig_size, metadata.comp_size, 0)) != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
ret = CL_SUCCESS;
if (metadata.filename)
free(metadata.filename);
continue;
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
ret = cli_unarj_extract_file(dir, &metadata);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ret != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("ARJ: cli_unarj_extract_file Error: %s\n", cl_strerror(ret));
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (metadata.ofd >= 0) {
if (lseek(metadata.ofd, 0, SEEK_SET) == -1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("ARJ: call to lseek() failed\n");
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = cli_magic_scan_desc(metadata.ofd, NULL, ctx, metadata.filename, LAYER_ATTRIBUTES_NONE);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(metadata.ofd);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (ret != CL_SUCCESS) {
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (metadata.filename) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(metadata.filename);
metadata.filename = NULL;
}
} while (ret == CL_SUCCESS);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!ctx->engine->keeptmp) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_rmdirs(dir);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
if (NULL != dir) {
free(dir);
}
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (metadata.filename) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(metadata.filename);
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
}
cli_dbgmsg("ARJ: Exit code: %d\n", ret);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (ret == CL_BREAK) {
ret = CL_SUCCESS;
}
add ARJ and SFX-ARJ support git-svn: trunk@3138
2007-07-11 10:14:08 +00:00
return ret;
}
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
static cl_error_t cli_scangzip_with_zib_from_the_80s(cli_ctx *ctx, unsigned char *buff)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
{
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
int fd;
cl_error_t ret;
size_t outsize = 0;
int bytes;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fmap_t *map = ctx->fmap;
fix gzip handler
2010-02-09 16:36:14 +01:00
char *tmpname;
gzFile gz;
cid #11072,#11138,#11329,#11330,#11331,#11332,#11333
2013-02-19 15:56:26 -05:00
ret = fmap_fd(map);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (ret < 0)
return CL_EDUP;
cid #11072,#11138,#11329,#11330,#11331,#11332,#11333
2013-02-19 15:56:26 -05:00
fd = dup(ret);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (fd < 0)
return CL_EDUP;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!(gz = gzdopen(fd, "rb"))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
return CL_EOPEN;
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_gentempfd(ctx->this_layer_tmpdir, &tmpname, &fd)) != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("GZip: Can't generate temporary file.\n");
gzclose(gz);
close(fd);
return ret;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while ((bytes = gzread(gz, buff, FILEBUFF)) > 0) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
outsize += bytes;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (cli_checklimits("GZip", ctx, outsize, 0, 0) != CL_SUCCESS)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
break;
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
if (cli_writen(fd, buff, (size_t)bytes) != (size_t)bytes) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
gzclose(gz);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
free(tmpname);
return CL_EWRITE;
}
fix gzip handler
2010-02-09 16:36:14 +01:00
}
gzclose(gz);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != (ret = cli_magic_scan_desc(fd, tmpname, ctx, NULL, LAYER_ATTRIBUTES_NONE))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
(void)cli_unlink(tmpname);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
free(tmpname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return ret;
fix gzip handler
2010-02-09 16:36:14 +01:00
}
close(fd);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
ret = CL_EUNLINK;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
}
fix gzip handler
2010-02-09 16:36:14 +01:00
free(tmpname);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scangzip(cli_ctx *ctx)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int fd;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
unsigned char buff[FILEBUFF];
char *tmpname;
z_stream z;
size_t at = 0, outsize = 0;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fmap_t *map = ctx->fmap;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
SCANBUFF -> FILEBUFF fix git-svn: trunk@107
2003-11-09 19:26:44 +00:00
cli_dbgmsg("in cli_scangzip()\n");
gunzip to fmap
2009-09-01 23:33:17 +02:00
memset(&z, 0, sizeof(z));
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if ((ret = inflateInit2(&z, MAX_WBITS + 16)) != Z_OK) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("GZip: InflateInit failed: %d\n", ret);
return cli_scangzip_with_zib_from_the_80s(ctx, buff);
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_gentempfd(ctx->this_layer_tmpdir, &tmpname, &fd)) != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("GZip: Can't generate temporary file.\n");
inflateEnd(&z);
return ret;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while (at < map->len) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
unsigned int bytes = MIN(map->len - at, map->pgsz);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!(z.next_in = (void *)fmap_need_off_once(map, at, bytes))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("GZip: Can't read %u bytes @ %lu.\n", bytes, (long unsigned)at);
inflateEnd(&z);
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
free(tmpname);
return CL_EREAD;
}
at += bytes;
z.avail_in = bytes;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
do {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
int inf;
z.avail_out = sizeof(buff);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
z.next_out = buff;
inf = inflate(&z, Z_NO_FLUSH);
if (inf != Z_OK && inf != Z_STREAM_END && inf != Z_BUF_ERROR) {
if (sizeof(buff) == z.avail_out) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("GZip: Bad stream, nothing in output buffer.\n");
at = map->len;
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("GZip: Bad stream, data in output buffer.\n");
/* no break yet, flush extracted bytes to file */
}
}
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
if (cli_writen(fd, buff, sizeof(buff) - z.avail_out) == (size_t)-1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
inflateEnd(&z);
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
free(tmpname);
return CL_EWRITE;
}
outsize += sizeof(buff) - z.avail_out;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (cli_checklimits("GZip", ctx, outsize, 0, 0) != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
at = map->len;
break;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (inf == Z_STREAM_END) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
at -= z.avail_in;
inflateReset(&z);
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else if (inf != Z_OK && inf != Z_BUF_ERROR) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
at = map->len;
break;
}
} while (z.avail_out == 0);
}
inflateEnd(&z);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != (ret = cli_magic_scan_desc(fd, tmpname, ctx, NULL, LAYER_ATTRIBUTES_NONE))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
}
free(tmpname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return ret;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
various cleanups; fix bb#577 (move cli_* out of clamav.h) git-svn: trunk@3196
2007-08-31 19:55:09 +00:00
close(fd);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
if (cli_unlink(tmpname))
ret = CL_EUNLINK;
gunzip to fmap
2009-09-01 23:33:17 +02:00
free(tmpname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
return ret;
}
#ifdef NOBZ2PREFIX
fmapify cli_scanbzip
2011-06-10 21:22:46 +03:00
#define BZ2_bzDecompressInit bzDecompressInit
#define BZ2_bzDecompress bzDecompress
#define BZ2_bzDecompressEnd bzDecompressEnd
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
#endif
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanbzip(cli_ctx *ctx)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int fd, rc;
bb12332 - fix segfault when scanning bz2 compressed iso9660s by limiting total page prefaulting to the 4GB max
2020-01-30 09:15:44 -08:00
uint64_t size = 0;
fmapify cli_scanbzip
2011-06-10 21:22:46 +03:00
char *tmpname;
bz_stream strm;
size_t off = 0;
size_t avail;
char buf[FILEBUFF];
memset(&strm, 0, sizeof(strm));
Error reporting touchup for max-filesize
2023-08-23 15:26:50 -07:00
strm.next_out = buf;
fmapify cli_scanbzip
2011-06-10 21:22:46 +03:00
strm.avail_out = sizeof(buf);
Error reporting touchup for max-filesize
2023-08-23 15:26:50 -07:00
rc = BZ2_bzDecompressInit(&strm, 0, 0);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (BZ_OK != rc) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Bzip: DecompressInit failed: %d\n", rc);
return CL_EOPEN;
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_gentempfd(ctx->this_layer_tmpdir, &tmpname, &fd))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Bzip: Can't generate temporary file.\n");
BZ2_bzDecompressEnd(&strm);
return ret;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
do {
if (!strm.avail_in) {
Error reporting touchup for max-filesize
2023-08-23 15:26:50 -07:00
strm.next_in = (void *)fmap_need_off_once_len(ctx->fmap, off, FILEBUFF, &avail);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
strm.avail_in = avail;
off += avail;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!strm.avail_in) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Bzip: premature end of compressed stream\n");
break;
}
}
rc = BZ2_bzDecompress(&strm);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (BZ_OK != rc && BZ_STREAM_END != rc) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Bzip: decompress error: %d\n", rc);
break;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!strm.avail_out || BZ_STREAM_END == rc) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
size += sizeof(buf) - strm.avail_out;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cli_writen(fd, buf, sizeof(buf) - strm.avail_out) != sizeof(buf) - strm.avail_out) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Bzip: Can't write to file.\n");
BZ2_bzDecompressEnd(&strm);
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
}
free(tmpname);
return CL_EWRITE;
}
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (cli_checklimits("Bzip", ctx, size, 0, 0) != CL_SUCCESS)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
break;
Error reporting touchup for max-filesize
2023-08-23 15:26:50 -07:00
strm.next_out = buf;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
strm.avail_out = sizeof(buf);
}
fmapify cli_scanbzip
2011-06-10 21:22:46 +03:00
} while (BZ_STREAM_END != rc);
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
fmapify cli_scanbzip
2011-06-10 21:22:46 +03:00
BZ2_bzDecompressEnd(&strm);
support ArchiveBlockMax in scan(g|b)zip() git-svn: trunk@1457
2005-04-06 22:48:06 +00:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != (ret = cli_magic_scan_desc(fd, tmpname, ctx, NULL, LAYER_ATTRIBUTES_NONE))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return CL_EUNLINK;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
free(tmpname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return ret;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
various cleanups; fix bb#577 (move cli_* out of clamav.h) git-svn: trunk@3196
2007-08-31 19:55:09 +00:00
close(fd);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
if (cli_unlink(tmpname))
ret = CL_EUNLINK;
fmapify cli_scanbzip
2011-06-10 21:22:46 +03:00
free(tmpname);
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanxz(cli_ctx *ctx)
XZ file type support
2013-10-08 17:17:44 -04:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int fd, rc;
XZ file type support
2013-10-08 17:17:44 -04:00
unsigned long int size = 0;
char *tmpname;
Silence compiler warnings.
2014-07-09 13:16:31 -04:00
struct CLI_XZ strm;
XZ file type support
2013-10-08 17:17:44 -04:00
size_t off = 0;
size_t avail;
Silence compiler warnings.
2014-07-09 13:16:31 -04:00
unsigned char *buf;
XZ file type support
2013-10-08 17:17:44 -04:00
Optimization: replace limited allocation calls There are a large number of allocations for fix sized buffers using the `cli_malloc` and `cli_calloc` calls that check if the requested size is larger than our allocation threshold for allocations based on untrusted input. These allocations will *always* be higher than the threshold, so the extra stack frame and check for these calls is a waste of CPU. This commit replaces needless calls with A -> B: - cli_malloc -> malloc - cli_calloc -> calloc - CLI_MALLOC -> MALLOC - CLI_CALLOC -> CALLOC I also noticed that our MPOOL_MALLOC / MPOOL_CALLOC are not limited by the max-allocation threshold, when MMAP is found/enabled. But the alternative was set to cli_malloc / cli_calloc when disabled. I changed those as well. I didn't change the cli_realloc/2 calls because our version of realloc not only implements a threshold but also stabilizes the undefined behavior in realloc to protect against accidental double-free's. It may be worth implementing a cli_realloc that doesn't have the threshold built-in, however, so as to allow reallocaitons for things like buffers for loading signatures, which aren't subject to the same concern as allocations for scanning possible malware. There was one case in mbox.c where I changed MALLOC -> CLI_MALLOC, because it appears to be allocating based on untrusted input.
2022-05-08 14:59:09 -07:00
buf = malloc(CLI_XZ_OBUF_SIZE);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (buf == NULL) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanxz: nomemory for decompress buffer.\n");
use heap for XZ decompress buffer
2013-10-09 15:41:55 -04:00
return CL_EMEM;
}
Silence compiler warnings.
2014-07-09 13:16:31 -04:00
memset(&strm, 0x00, sizeof(struct CLI_XZ));
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
strm.next_out = buf;
use heap for XZ decompress buffer
2013-10-09 15:41:55 -04:00
strm.avail_out = CLI_XZ_OBUF_SIZE;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
rc = cli_XzInit(&strm);
if (rc != XZ_RESULT_OK) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanxz: DecompressInit failed: %i\n", rc);
use heap for XZ decompress buffer
2013-10-09 15:41:55 -04:00
free(buf);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return CL_EOPEN;
XZ file type support
2013-10-08 17:17:44 -04:00
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_gentempfd(ctx->this_layer_tmpdir, &tmpname, &fd))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanxz: Can't generate temporary file.\n");
cli_XzShutdown(&strm);
use heap for XZ decompress buffer
2013-10-09 15:41:55 -04:00
free(buf);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return ret;
XZ file type support
2013-10-08 17:17:44 -04:00
}
cli_dbgmsg("cli_scanxz: decompressing to file %s\n", tmpname);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
do {
XZ file type support
2013-10-08 17:17:44 -04:00
/* set up input buffer */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!strm.avail_in) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
strm.next_in = (void *)fmap_need_off_once_len(ctx->fmap, off, CLI_XZ_IBUF_SIZE, &avail);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
strm.avail_in = avail;
off += avail;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!strm.avail_in) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanxz: premature end of compressed stream\n");
XZ file type support
2013-10-08 17:17:44 -04:00
ret = CL_EFORMAT;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
goto xz_exit;
}
}
XZ file type support
2013-10-08 17:17:44 -04:00
/* xz decompress a chunk */
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
rc = cli_XzDecode(&strm);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (XZ_RESULT_OK != rc && XZ_STREAM_END != rc) {
if (rc == XZ_DIC_HEURISTIC) {
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
ret = cli_append_potentially_unwanted(ctx, "Heuristics.XZ.DicSizeLimit");
bb11625 - adding heuristic check for XZ dictionary size evasion
2017-06-19 15:41:17 -04:00
goto xz_exit;
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanxz: decompress error: %d\n", rc);
XZ file type support
2013-10-08 17:17:44 -04:00
ret = CL_EFORMAT;
goto xz_exit;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
// cli_dbgmsg("cli_scanxz: xz decompressed %li of %li available bytes\n",
// avail - strm.avail_in, avail);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
XZ file type support
2013-10-08 17:17:44 -04:00
/* write decompress buffer */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!strm.avail_out || rc == XZ_STREAM_END) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
size_t towrite = CLI_XZ_OBUF_SIZE - strm.avail_out;
size += towrite;
XZ file type support
2013-10-08 17:17:44 -04:00
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
// cli_dbgmsg("Writing %li bytes to XZ decompress temp file(%li byte total)\n",
// towrite, size);
XZ file type support
2013-10-08 17:17:44 -04:00
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
if (cli_writen(fd, buf, towrite) != towrite) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanxz: Can't write to file.\n");
XZ file type support
2013-10-08 17:17:44 -04:00
ret = CL_EWRITE;
goto xz_exit;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (cli_checklimits("cli_scanxz", ctx, size, 0, 0) != CL_SUCCESS) {
XZ file type support
2013-10-08 17:17:44 -04:00
cli_warnmsg("cli_scanxz: decompress file size exceeds limits - "
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
"only scanning %li bytes\n",
size);
break;
XZ file type support
2013-10-08 17:17:44 -04:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
strm.next_out = buf;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
strm.avail_out = CLI_XZ_OBUF_SIZE;
}
XZ file type support
2013-10-08 17:17:44 -04:00
} while (XZ_STREAM_END != rc);
/* scan decompressed file */
Fix issue reporting all Heuristic/PUA matches in allmatch mode When potentially unwanted indicators are found, they are not reported right away, except in heuristic-precedence mode. At the end of the scan, we then report all of the heuristic/PUA alerts. In my initial overhaul of the allmatch mode, I introduced an issue where it is never reporting more than 1 heuristic/PUA alert. This commit fixes that by reporting all potentially-unwanted indicators at the end. Note that instead of using `cli_virus_found_cb()` which would report the top layer file descriptor as well in the callback, I'm reporting `-1` for the file descriptor instead, because we don't know when the alert was added. If it was at a deeper layer in the file, we would not longer have access to it at the end of the scan. Also removed excessive debug statements '... infected with ...' that use `cli_get_last_virus()` because it doesn't really reflect all of the matches in allmatch mode, because the match would've been reported previously (so it is excessive), and because I don't like the word 'infected'. :)
2022-08-16 19:13:05 -07:00
ret = cli_magic_scan_desc(fd, tmpname, ctx, NULL, LAYER_ATTRIBUTES_NONE);
XZ file type support
2013-10-08 17:17:44 -04:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
xz_exit:
XZ file type support
2013-10-08 17:17:44 -04:00
cli_XzShutdown(&strm);
close(fd);
Fix issue reporting all Heuristic/PUA matches in allmatch mode When potentially unwanted indicators are found, they are not reported right away, except in heuristic-precedence mode. At the end of the scan, we then report all of the heuristic/PUA alerts. In my initial overhaul of the allmatch mode, I introduced an issue where it is never reporting more than 1 heuristic/PUA alert. This commit fixes that by reporting all potentially-unwanted indicators at the end. Note that instead of using `cli_virus_found_cb()` which would report the top layer file descriptor as well in the callback, I'm reporting `-1` for the file descriptor instead, because we don't know when the alert was added. If it was at a deeper layer in the file, we would not longer have access to it at the end of the scan. Also removed excessive debug statements '... infected with ...' that use `cli_get_last_virus()` because it doesn't really reflect all of the matches in allmatch mode, because the match would've been reported previously (so it is excessive), and because I don't like the word 'infected'. :)
2022-08-16 19:13:05 -07:00
if (!ctx->engine->keeptmp) {
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (cli_unlink(tmpname) && ret == CL_SUCCESS) {
XZ file type support
2013-10-08 17:17:44 -04:00
ret = CL_EUNLINK;
Fix issue reporting all Heuristic/PUA matches in allmatch mode When potentially unwanted indicators are found, they are not reported right away, except in heuristic-precedence mode. At the end of the scan, we then report all of the heuristic/PUA alerts. In my initial overhaul of the allmatch mode, I introduced an issue where it is never reporting more than 1 heuristic/PUA alert. This commit fixes that by reporting all potentially-unwanted indicators at the end. Note that instead of using `cli_virus_found_cb()` which would report the top layer file descriptor as well in the callback, I'm reporting `-1` for the file descriptor instead, because we don't know when the alert was added. If it was at a deeper layer in the file, we would not longer have access to it at the end of the scan. Also removed excessive debug statements '... infected with ...' that use `cli_get_last_virus()` because it doesn't really reflect all of the matches in allmatch mode, because the match would've been reported previously (so it is excessive), and because I don't like the word 'infected'. :)
2022-08-16 19:13:05 -07:00
}
}
XZ file type support
2013-10-08 17:17:44 -04:00
free(tmpname);
use heap for XZ decompress buffer
2013-10-09 15:41:55 -04:00
free(buf);
XZ file type support
2013-10-08 17:17:44 -04:00
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanszdd(cli_ctx *ctx)
support files compressed with compress.exe git-svn: trunk@535
2004-05-02 00:51:01 +00:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int ofd;
cl_error_t ret;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
char *tmpname;
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
update git-svn: trunk@1610
2005-06-11 20:27:59 +00:00
cli_dbgmsg("in cli_scanszdd()\n");
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_gentempfd(ctx->this_layer_tmpdir, &tmpname, &ofd))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("MSEXPAND: Can't generate temporary file/descriptor\n");
return ret;
support files compressed with compress.exe git-svn: trunk@535
2004-05-02 00:51:01 +00:00
}
fmapify msexpand
2011-06-10 19:09:38 +02:00
ret = cli_msexpand(ctx, ofd);
support files compressed with compress.exe git-svn: trunk@535
2004-05-02 00:51:01 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ret != CL_SUCCESS) { /* CL_VIRUS or some error */
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(ofd);
if (!ctx->engine->keeptmp)
if (cli_unlink(tmpname))
ret = CL_EUNLINK;
free(tmpname);
return ret;
support files compressed with compress.exe git-svn: trunk@535
2004-05-02 00:51:01 +00:00
}
rewritten decompressor for mscompress git-svn: trunk@3416
2007-12-13 23:18:03 +00:00
cli_dbgmsg("MSEXPAND: Decompressed into %s\n", tmpname);
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_desc(ofd, tmpname, ctx, NULL, LAYER_ATTRIBUTES_NONE);
rewritten decompressor for mscompress git-svn: trunk@3416
2007-12-13 23:18:03 +00:00
close(ofd);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
if (cli_unlink(tmpname))
ret = CL_EUNLINK;
free(tmpname);
rewritten decompressor for mscompress git-svn: trunk@3416
2007-12-13 23:18:03 +00:00
support files compressed with compress.exe git-svn: trunk@535
2004-05-02 00:51:01 +00:00
return ret;
}
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
static cl_error_t vba_scandata(const unsigned char *data, size_t len, cli_ctx *ctx)
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
{
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
cl_error_t ret = CL_SUCCESS;
struct cli_matcher *generic_ac_root = ctx->engine->root[0];
struct cli_matcher *target_ac_root = ctx->engine->root[2];
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct cli_ac_data gmdata, tmdata;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
bool gmdata_initialized = false;
bool tmdata_initialized = false;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct cli_ac_data *mdata[2];
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
bool must_pop_stack = false;
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cl_fmap_t *new_map = NULL;
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
if ((ret = cli_ac_initdata(&tmdata, target_ac_root->ac_partsigs, target_ac_root->ac_lsigs, target_ac_root->ac_reloff_num, CLI_DEFAULT_AC_TRACKLEN))) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
goto done;
}
tmdata_initialized = true;
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
if ((ret = cli_ac_initdata(&gmdata, generic_ac_root->ac_partsigs, generic_ac_root->ac_lsigs, generic_ac_root->ac_reloff_num, CLI_DEFAULT_AC_TRACKLEN))) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
goto done;
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
gmdata_initialized = true;
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
mdata[0] = &tmdata;
mdata[1] = &gmdata;
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
ret = cli_scan_buff(data, len, 0, ctx, CL_TYPE_MSOLE2, mdata);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
goto done;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
* Evaluate logical & yara rules given the new matches to see if anything alerts.
*/
new_map = fmap_open_memory(data, len, NULL);
if (new_map == NULL) {
cli_dbgmsg("Failed to create fmap for evaluating logical/yara rules after call to cli_scan_buff()\n");
ret = CL_EMEM;
goto done;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = cli_recursion_stack_push(ctx, new_map, CL_TYPE_MSOLE2, true, LAYER_ATTRIBUTES_NONE); /* Perform exp_eval with child fmap */
if (CL_SUCCESS != ret) {
cli_dbgmsg("Failed to scan fmap.\n");
goto done;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
must_pop_stack = true;
fix scan all segv handling file type MSOLE2
2012-11-30 11:16:47 -08:00
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_exp_eval(ctx, target_ac_root, &tmdata, NULL);
Improvements to adhere to scan time limit Error handling for bytecode sig evaluation, as well as logical and yara sig evaluation only consider CL_VIRUS or CL_SUCCESS and fail to consider terminal errors such as CL_ETIMEOUT. This commit fixes the error handling in those functions so we properly abort instead of continuing to evaluate more sigs. This commit also adds some a scan time limit checks: 1. shortly after a bytecode time limit exceeded. 2. after evaluating each lsig or yara sig. 3. when processing unzip central directory file headers.
2022-10-03 18:23:05 -07:00
if (CL_SUCCESS != ret) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
goto done;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_exp_eval(ctx, generic_ac_root, &gmdata, NULL);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
done:
if (must_pop_stack) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
(void)cli_recursion_stack_pop(ctx); /* Restore the parent fmap */
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (NULL != new_map) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(new_map);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
if (tmdata_initialized) {
cli_ac_freedata(&tmdata);
}
if (gmdata_initialized) {
cli_ac_freedata(&gmdata);
}
return ret;
libclamav/scanners.c: use lsigs when scanning vba data (bb#3922)
2011-11-18 15:25:04 +01:00
}
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
/**
* Find a file in a directory tree.
* \param filename Name of the file to find
* \param dir Directory path where to find the file
* \param A pointer to the string to store the result into
* \param Size of the string to store the result in
*/
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t find_file(const char *filename, const char *dir, char *result, size_t result_size)
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
{
DIR *dd;
struct dirent *dent;
char fullname[PATH_MAX];
cl_error_t ret;
size_t len;
STATBUF statbuf;
if (!result) {
return CL_ENULLARG;
}
if ((dd = opendir(dir)) != NULL) {
while ((dent = readdir(dd))) {
if (dent->d_ino) {
if (strcmp(dent->d_name, ".") != 0 && strcmp(dent->d_name, "..") != 0) {
snprintf(fullname, sizeof(fullname), "%s" PATHSEP "%s", dir, dent->d_name);
fullname[sizeof(fullname) - 1] = '\0';
/* stat the file */
if (LSTAT(fullname, &statbuf) != -1) {
if (S_ISDIR(statbuf.st_mode) && !S_ISLNK(statbuf.st_mode)) {
ret = find_file(filename, fullname, result, result_size);
if (ret == CL_SUCCESS) {
closedir(dd);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
} else if (S_ISREG(statbuf.st_mode)) {
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
if (strcmp(dent->d_name, filename) == 0) {
Windows: Fix C/Rust FFI compat issue + Windows compile warnings Primarily this commit fixes an issue with the size of the parameters passed to cli_checklimits(). The parameters were "unsigned long", which varies in size depending on platform. I've switched them to uint64_t / u64. While working on this, I observed some concerning warnigns on Windows, and some less serious ones, primarily regarding inconsistencies with `const` parameters. Finally, in `scanmem.c`, there is a warning regarding use of `wchar_t *` with `GetModuleFileNameEx()` instead of `GetModuleFileNameExW()`. This made me realize this code assumes we're not defining `UNICODE`, which would have such macros use the 'A' variant. I have fixed it the best I can, although I'm still a little uncomfortable with some of this code that uses `char` or `wchar_t` instead of TCHAR. I also remove the `if (GetModuleFileNameEx) {` conditional, because this macro/function will always be defined. The original code was checking a function pointer, and so this was a bug when integrating into ClamAV. Regarding the changes to `rijndael.c`, I found that this module assumes `unsigned long` == 32bits. It does not. I have corrected it to use `uint32_t`.
2024-03-20 12:21:40 -04:00
len = MIN(strlen(dir) + 1, result_size);
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
memcpy(result, dir, len);
result[len - 1] = '\0';
closedir(dd);
return CL_SUCCESS;
}
}
}
}
}
}
closedir(dd);
}
return CL_EOPEN;
}
/**
* Scan an OLE directory for a VBA project.
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
* Contrary to cli_ole2_tempdir_scan_vba, this function uses the dir file to locate VBA modules.
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
*/
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
static cl_error_t cli_ole2_tempdir_scan_vba_new(const char *dir, cli_ctx *ctx, struct uniq *U, int *has_macros)
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t ret = CL_SUCCESS;
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
uint32_t hashcnt = 0;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
char *hash = NULL;
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
char path[PATH_MAX];
char filename[PATH_MAX];
clang format Running clang format on codebase
2023-04-18 14:29:30 -04:00
int tempfd = -1;
char *tempfile = NULL;
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
if (CL_SUCCESS != (ret = uniq_get(U, "dir", 3, &hash, &hashcnt))) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("cli_ole2_tempdir_scan_vba_new: uniq_get('dir') failed with ret code (%d)!\n", ret);
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
return ret;
}
while (hashcnt) {
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
// Find the directory containing the extracted dir file. This is complicated
// because ClamAV doesn't use the file names from the OLE file, but temporary names,
// and we have neither the complete path of the dir file in the OLE container,
// nor the mapping of the temporary directory names to their OLE names.
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
snprintf(filename, sizeof(filename), "%s_%u", hash, hashcnt);
filename[sizeof(filename) - 1] = '\0';
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (CL_SUCCESS == find_file(filename, dir, path, sizeof(path))) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba_new: Found dir file: %s\n", path);
2287 - fix ole2 vba temp file leak Previous behaviour would remove temp files by deleting the subdirectory This caused issues in cases (on Windows) where subdirectories aren't created due to performance concerns This commit removes tempfiles individually if keeptemp is off Original patch authored by Thomas Vy
2023-04-12 16:08:20 -04:00
if ((ret = cli_vba_readdir_new(ctx, path, U, hash, hashcnt, &tempfd, has_macros, &tempfile)) != CL_SUCCESS) {
Add explicit log level parameter to application logging API * Added loglevel parameter to logg() * Fix logg and mprintf internals with new loglevels * Update all logg calls to set loglevel * Update all mprintf calls to set loglevel * Fix hidden logg calls * Executed clam-format
2022-02-16 00:13:55 +01:00
// FIXME: Since we only know the stream name of the OLE2 stream, but not its path inside the
// OLE2 archive, we don't know if we have the right file. The only thing we can do is
// iterate all of them until one succeeds.
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("cli_ole2_tempdir_scan_vba_new: Failed to read dir from %s, trying others (error: %s (%d))\n", path, cl_strerror(ret), (int)ret);
Fix small leak in VBA parser Fixed a leak introduced by a recent bug fix in the VBA parser. Fixes: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=58301
2023-04-25 20:33:55 -07:00
if (tempfile) {
if (!ctx->engine->keeptmp) {
remove(tempfile);
}
free(tempfile);
tempfile = NULL;
}
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
ret = CL_SUCCESS;
hashcnt--;
continue;
}
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (*has_macros && SCAN_COLLECT_METADATA && (ctx->this_layer_metadata_json != NULL)) {
cli_jsonbool(ctx->this_layer_metadata_json, "HasMacros", 1);
json_object *macro_languages = cli_jsonarray(ctx->this_layer_metadata_json, "MacroLanguages");
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
if (macro_languages) {
cli_jsonstr(macro_languages, NULL, "VBA");
} else {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("[cli_ole2_tempdir_scan_vba_new] Failed to add \"VBA\" entry to MacroLanguages JSON array\n");
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
}
}
Remove libjson-c dead code As of ClamAV 0.105, libjson-c is required. There is also no option to disable libjson-c support. This commit removes the dead code associated with the old build option.
2024-03-25 13:01:46 -04:00
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
if (SCAN_HEURISTIC_MACROS && *has_macros) {
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
ret = cli_append_potentially_unwanted(ctx, "Heuristics.OLE2.ContainsMacros.VBA");
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
if (ret == CL_VIRUS) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
goto done;
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
}
}
/*
* Now rewind the extracted vba-project output FD and scan it!
*/
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
if (lseek(tempfd, 0, SEEK_SET) != 0) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("cli_ole2_tempdir_scan_vba_new: Failed to seek to beginning of temporary VBA project file\n");
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
ret = CL_ESEEK;
goto done;
}
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
ret = cli_scan_desc(tempfd, ctx, CL_TYPE_SCRIPT, false, NULL, AC_SCAN_VIR, NULL, "extracted-vba-project", tempfile, LAYER_ATTRIBUTES_NORMALIZED);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
goto done;
}
clang format Running clang format on codebase
2023-04-18 14:29:30 -04:00
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
close(tempfd);
tempfd = -1;
2287 - fix ole2 vba temp file leak Previous behaviour would remove temp files by deleting the subdirectory This caused issues in cases (on Windows) where subdirectories aren't created due to performance concerns This commit removes tempfiles individually if keeptemp is off Original patch authored by Thomas Vy
2023-04-12 16:08:20 -04:00
if (tempfile) {
if (!ctx->engine->keeptmp) {
remove(tempfile);
}
free(tempfile);
tempfile = NULL;
}
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
}
hashcnt--;
}
done:
if (tempfd != -1) {
close(tempfd);
tempfd = -1;
}
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
2287 - fix ole2 vba temp file leak Previous behaviour would remove temp files by deleting the subdirectory This caused issues in cases (on Windows) where subdirectories aren't created due to performance concerns This commit removes tempfiles individually if keeptemp is off Original patch authored by Thomas Vy
2023-04-12 16:08:20 -04:00
if (tempfile) {
if (!ctx->engine->keeptmp) {
remove(tempfile);
}
free(tempfile);
tempfile = NULL;
}
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
return ret;
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
/**
* @brief find the summary information files and write out the meta to the JSON.
*
* @param dir The directory containing ole2 temp files
* @param ctx The scan context
* @param U The unique structure indicating while files exist in the directory
* @return cl_error_t
*/
static cl_error_t cli_ole2_tempdir_scan_summary(const char *dir, cli_ctx *ctx, struct uniq *U)
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cl_error_t ret;
char summary_filename[1024];
char *hash;
uint32_t hashcnt = 0;
if (CL_SUCCESS != (ret = uniq_get(U, "_5_summaryinformation", 21, &hash, &hashcnt))) {
cli_dbgmsg("cli_ole2_tempdir_scan_summary: uniq_get('_5_summaryinformation') failed with ret code (%d)!\n", ret);
status = ret;
goto done;
}
while (hashcnt) {
int fd = -1;
snprintf(summary_filename, sizeof(summary_filename), "%s" PATHSEP "%s_%u", dir, hash, hashcnt);
summary_filename[sizeof(summary_filename) - 1] = '\0';
fd = open(summary_filename, O_RDONLY | O_BINARY);
if (fd >= 0) {
cli_dbgmsg("cli_ole2_tempdir_scan_summary: detected a '_5_summaryinformation' stream\n");
/* JSONOLE2 - what to do if something breaks? */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
cli_ole2_summary_json(ctx, fd, 0, summary_filename);
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
close(fd);
}
hashcnt--;
}
if (CL_SUCCESS != (ret = uniq_get(U, "_5_documentsummaryinformation", 29, &hash, &hashcnt))) {
cli_dbgmsg("cli_ole2_tempdir_scan_summary: uniq_get('_5_documentsummaryinformation') failed with ret code (%d)!\n", ret);
status = ret;
goto done;
}
while (hashcnt) {
int fd = -1;
snprintf(summary_filename, sizeof(summary_filename), "%s" PATHSEP "%s_%u", dir, hash, hashcnt);
summary_filename[sizeof(summary_filename) - 1] = '\0';
fd = open(summary_filename, O_RDONLY | O_BINARY);
if (fd >= 0) {
cli_dbgmsg("cli_ole2_tempdir_scan_summary: detected a '_5_documentsummaryinformation' stream\n");
/* JSONOLE2 - what to do if something breaks? */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
cli_ole2_summary_json(ctx, fd, 1, summary_filename);
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
close(fd);
}
hashcnt--;
}
done:
return status;
}
/**
* @brief Check the ole2 temp directory for embedded OLE objects
*
* @param dir The ole2 temp directory
* @param ctx The scan context
* @param U The uniq structure which recors what files are in the temp directory
* @return cl_error_t
*/
static cl_error_t cli_ole2_tempdir_scan_embedded_ole10(const char *dir, cli_ctx *ctx, struct uniq *U)
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cl_error_t ret;
char ole10_filename[1024];
char *hash;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
uint32_t hashcnt = 0;
int fd = -1;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
/* Check directory for embedded OLE objects */
if (CL_SUCCESS != (ret = uniq_get(U, "_1_ole10native", 14, &hash, &hashcnt))) {
cli_dbgmsg("cli_ole2_tempdir_scan_embedded_ole10: uniq_get('_1_ole10native') failed with ret code (%d)!\n", ret);
status = ret;
goto done;
}
while (hashcnt) {
snprintf(ole10_filename, sizeof(ole10_filename), "%s" PATHSEP "%s_%u", dir, hash, hashcnt);
ole10_filename[sizeof(ole10_filename) - 1] = '\0';
fd = open(ole10_filename, O_RDONLY | O_BINARY);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (fd < 0) {
hashcnt--;
continue;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = cli_scan_ole10(fd, ctx);
if (CL_SUCCESS != ret) {
status = ret;
goto done;
}
close(fd);
fd = -1;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
hashcnt--;
}
done:
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (fd >= 0) {
close(fd);
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
}
return status;
}
static cl_error_t cli_ole2_tempdir_scan_vba(const char *dir, cli_ctx *ctx, struct uniq *U, int *has_macros)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cl_error_t status = CL_SUCCESS;
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
cl_error_t ret;
Fix new issues identified by Coverity 298485: Fix possible fd leaks. 298486: Fix possible use-after-free.
2020-08-12 18:14:39 -07:00
int i, j;
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
size_t data_len;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
vba_project_t *vba_project;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
char *fullname = NULL;
char vbaname[1024];
unsigned char *data = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
char *hash;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
uint32_t hashcnt = 0;
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
int fd = -1;
int proj_contents_fd = -1;
char *proj_contents_fname = NULL;
if (CL_SUCCESS != (status = uniq_get(U, "_vba_project", 12, NULL, &hashcnt))) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: uniq_get('_vba_project') failed with ret code (%d)!\n", status);
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
goto done;
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
}
while (hashcnt) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (!(vba_project = (vba_project_t *)cli_vba_readdir(dir, U, hashcnt))) {
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
hashcnt--;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
continue;
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
for (i = 0; i < vba_project->count; i++) {
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
for (j = 1; (unsigned int)j <= vba_project->colls[i]; j++) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
snprintf(vbaname, 1024, "%s" PATHSEP "%s_%u", vba_project->dir, vba_project->name[i], j);
vbaname[sizeof(vbaname) - 1] = '\0';
Fix new issues identified by Coverity 298485: Fix possible fd leaks. 298486: Fix possible use-after-free.
2020-08-12 18:14:39 -07:00
fd = open(vbaname, O_RDONLY | O_BINARY);
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
if (fd == -1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
continue;
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("cli_ole2_tempdir_scan_vba: Decompress VBA project '%s_%u'\n", vba_project->name[i], j);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
data = (unsigned char *)cli_vba_inflate(fd, vba_project->offset[i], &data_len);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fd = -1;
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
*has_macros = *has_macros + 1;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (NULL != data) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* cli_dbgmsg("Project content:\n%s", data); */
if (ctx->scanned)
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
*ctx->scanned += data_len;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ctx->engine->keeptmp) {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (CL_SUCCESS != (status = cli_gentempfd(ctx->this_layer_tmpdir, &proj_contents_fname, &proj_contents_fd))) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_warnmsg("WARNING: VBA project '%s_%u' cannot be dumped to file\n", vba_project->name[i], j);
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (cli_writen(proj_contents_fd, data, data_len) != data_len) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_warnmsg("WARNING: VBA project '%s_%u' failed to write to file\n", vba_project->name[i], j);
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
status = CL_EWRITE;
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
close(proj_contents_fd);
proj_contents_fd = -1;
Do not report timeout as error Because of an oversight, the `ctx->abort_scan` flag was propagating timeout and perhaps other error types to the final verdict. Fixed this in the `result_should_goto_done()` function. Some non-fatal errors (notably CL_ETIMEOUT) may appear between the call to the magic-scan, and the end of scan_common(). We need to filter these status results at this level as well to prevent this from happening. I also fixed a warning accidentally introduced earlier caused by trying to use a string after free'ing it and setting it to NULL. And I explicitly set the status to CL_SUCCESS if the cache check comes back clean, rather than trusting that no one had messed with the `ret` variable since it was let set (to CL_SUCCESS). It's more readable and less prone to future bugs.
2022-09-21 12:22:54 -07:00
cli_dbgmsg("cli_ole2_tempdir_scan_vba: VBA project '%s_%u' dumped to %s\n", vba_project->name[i], j, proj_contents_fname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
free(proj_contents_fname);
proj_contents_fname = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = vba_scandata(data, data_len, ctx);
if (CL_SUCCESS != status) {
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(data);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
data = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
}
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
cli_free_vba_project(vba_project);
vba_project = NULL;
hashcnt--;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != (status = uniq_get(U, "powerpoint document", 19, &hash, &hashcnt))) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: uniq_get('powerpoint document') failed with ret code (%d)!\n", status);
goto done;
}
while (hashcnt) {
snprintf(vbaname, 1024, "%s" PATHSEP "%s_%u", dir, hash, hashcnt);
vbaname[sizeof(vbaname) - 1] = '\0';
Fix new issues identified by Coverity 298485: Fix possible fd leaks. 298486: Fix possible use-after-free.
2020-08-12 18:14:39 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fd = open(vbaname, O_RDONLY | O_BINARY);
if (fd == -1) {
hashcnt--;
continue;
}
Fix new issues identified by Coverity 298485: Fix possible fd leaks. 298486: Fix possible use-after-free.
2020-08-12 18:14:39 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fullname = cli_ppt_vba_read(fd, ctx);
if (NULL != fullname) {
status = cli_magic_scan_dir(fullname, ctx, LAYER_ATTRIBUTES_NONE);
if (CL_SUCCESS != status) {
goto done;
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
}
Code review fixes Exit early from VBA scanning loop if virus found. Add VBA/XLM suffix to ContainsMacros heuristics. Fix setting status code for error and virus conditions. Increment/decrement recursion counter when scanning vba dir.
2020-08-10 14:57:28 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!ctx->engine->keeptmp) {
cli_rmdirs(fullname);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
free(fullname);
fullname = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
close(fd);
fd = -1;
hashcnt--;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != (status = uniq_get(U, "worddocument", 12, &hash, &hashcnt))) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: uniq_get('worddocument') failed with ret code (%d)!\n", status);
goto done;
}
while (hashcnt) {
snprintf(vbaname, sizeof(vbaname), "%s" PATHSEP "%s_%u", dir, hash, hashcnt);
vbaname[sizeof(vbaname) - 1] = '\0';
fd = open(vbaname, O_RDONLY | O_BINARY);
if (fd == -1) {
hashcnt--;
continue;
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
}
Fix new issues identified by Coverity 298485: Fix possible fd leaks. 298486: Fix possible use-after-free.
2020-08-12 18:14:39 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!(vba_project = (vba_project_t *)cli_wm_readdir(fd))) {
close(fd);
fd = -1;
hashcnt--;
continue;
}
Fix new issues identified by Coverity 298485: Fix possible fd leaks. 298486: Fix possible use-after-free.
2020-08-12 18:14:39 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
for (i = 0; i < vba_project->count; i++) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: Decompress WM project macro:%d key:%d length:%d\n", i, vba_project->key[i], vba_project->length[i]);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
data = (unsigned char *)cli_wm_decrypt_macro(fd, vba_project->offset[i], vba_project->length[i], vba_project->key[i]);
if (!data) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: WARNING: WM project '%s' macro %d decrypted to NULL\n", vba_project->name[i], i);
} else {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: Project content:\n%s", data);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (ctx->scanned) {
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
*ctx->scanned += vba_project->length[i];
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = vba_scandata(data, vba_project->length[i], ctx);
if (CL_SUCCESS != status) {
goto done;
}
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
free(data);
data = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
close(fd);
fd = -1;
cli_free_vba_project(vba_project);
vba_project = NULL;
hashcnt--;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
done:
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (*has_macros) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (SCAN_COLLECT_METADATA && (ctx->this_layer_metadata_json != NULL)) {
cli_jsonbool(ctx->this_layer_metadata_json, "HasMacros", 1);
json_object *macro_languages = cli_jsonarray(ctx->this_layer_metadata_json, "MacroLanguages");
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (macro_languages) {
cli_jsonstr(macro_languages, NULL, "VBA");
} else {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: Failed to add \"VBA\" entry to MacroLanguages JSON array\n");
}
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (SCAN_HEURISTIC_MACROS) {
ret = cli_append_potentially_unwanted(ctx, "Heuristics.OLE2.ContainsMacros.VBA");
if (ret == CL_VIRUS) {
status = ret;
}
}
clamd: add new option OLE2BlockMacros (requested by Mike)
2010-10-29 19:04:23 +02:00
}
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (proj_contents_fd >= 0) {
close(proj_contents_fd);
}
if (NULL != proj_contents_fname) {
free(proj_contents_fname);
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (NULL != data) {
free(data);
}
if (NULL != fullname) {
if (!ctx->engine->keeptmp) {
(void)cli_rmdirs(fullname);
}
free(fullname);
}
Coverity-401433: Fix file descriptor leak in VBA parser
2023-04-07 10:57:45 -07:00
if (fd >= 0) {
close(fd);
}
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
return status;
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
}
Fix XLM macro false positives in XLS docs containing images The XLS parsing code for extracting images and XLM macros is alerting with Heuristics.OLE2.ContainsMacros.XLM if any images are found if the --alert-macros (AlertOLE2Macros) option is enabled. This fix moves the alert logic before the XLM & image extraction logic where we know at least one macro exists, but before we try to extract it I've wrapped it with an "if (has_xlm)" condition. Resolves: - https://github.com/Cisco-Talos/clamav/issues/423 - https://bugzilla.clamav.net/show_bug.cgi?id=12844
2022-01-25 21:48:27 -08:00
static cl_error_t cli_ole2_tempdir_scan_for_xlm_and_images(const char *dir, cli_ctx *ctx, struct uniq *U)
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
char *hash = NULL;
uint32_t hashcnt = 0;
char STR_WORKBOOK[] = "workbook";
char STR_BOOK[] = "book";
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
if (CL_SUCCESS != (ret = uniq_get(U, STR_WORKBOOK, sizeof(STR_WORKBOOK) - 1, &hash, &hashcnt))) {
if (CL_SUCCESS != (ret = uniq_get(U, STR_BOOK, sizeof(STR_BOOK) - 1, &hash, &hashcnt))) {
Fix XLM macro false positives in XLS docs containing images The XLS parsing code for extracting images and XLM macros is alerting with Heuristics.OLE2.ContainsMacros.XLM if any images are found if the --alert-macros (AlertOLE2Macros) option is enabled. This fix moves the alert logic before the XLM & image extraction logic where we know at least one macro exists, but before we try to extract it I've wrapped it with an "if (has_xlm)" condition. Resolves: - https://github.com/Cisco-Talos/clamav/issues/423 - https://bugzilla.clamav.net/show_bug.cgi?id=12844
2022-01-25 21:48:27 -08:00
cli_dbgmsg("cli_ole2_tempdir_scan_for_xlm_and_images: uniq_get('%s') failed with ret code (%d)!\n", STR_BOOK, ret);
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
}
}
for (; hashcnt > 0; hashcnt--) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != (ret = cli_extract_xlm_macros_and_images(dir, ctx, hash, hashcnt))) {
Additional XLM parser error handling fixes Improve error handling for functions that read the XLM BIFF temp-files. Improve resource cleanup to alleviate Coverity false positive issue.
2020-07-16 13:39:47 -07:00
switch (ret) {
case CL_VIRUS:
case CL_EMEM:
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
Additional XLM parser error handling fixes Improve error handling for functions that read the XLM BIFF temp-files. Improve resource cleanup to alleviate Coverity false positive issue.
2020-07-16 13:39:47 -07:00
default:
Fix typos (no functional changes)
2023-11-26 15:01:19 -08:00
cli_dbgmsg("cli_ole2_tempdir_scan_for_xlm_and_images: An error occurred when parsing XLM BIFF temp file, skipping to next file.\n");
Additional XLM parser error handling fixes Improve error handling for functions that read the XLM BIFF temp-files. Improve resource cleanup to alleviate Coverity false positive issue.
2020-07-16 13:39:47 -07:00
}
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
}
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
done:
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
return ret;
}
libclamav: Add URI scanning support to PDF parser Threat Research requests scanning URIs in PDF files and adding them to the json report file. This change adds URI scanning support to the PDF parser, including support for object references to URIs in PDF files. Jira: CLAM-2588 Fix out-of-order references and other minor improvements. CLAM-2588, CLAM-2757
2025-04-07 16:50:09 -07:00
const char *const HTML_URIS_JSON_KEY = "URIs";
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
/* https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml */
const char *URI_LIST[] = {
"aaa://",
"aaas://",
"about://",
"acap://",
"acct://",
"acd://",
"acr://",
"adiumxtra://",
"adt://",
"afp://",
"afs://",
"aim://",
"amss://",
"android://",
"appdata://",
"apt://",
"ar://",
"ark://",
"at://",
"attachment://",
"aw://",
"barion://",
"bb://",
"beshare://",
"bitcoin://",
"bitcoincash://",
"blob://",
"bolo://",
"brid://",
"browserext://",
"cabal://",
"calculator://",
"callto://",
"cap://",
"cast://",
"casts://",
"chrome://",
"chrome-extension://",
"cid://",
"coap://",
"coap+tcp://",
"coap+ws://",
"coaps://",
"coaps+tcp://",
"coaps+ws://",
"com-eventbrite-attendee://",
"content://",
"content-type://",
"crid://",
"cstr://",
"cvs://",
"dab://",
"dat://",
"data://",
"dav://",
"dhttp://",
"diaspora://",
"dict://",
"did://",
"dis://",
"dlna-playcontainer://",
"dlna-playsingle://",
"dns://",
"dntp://",
"doi://",
"dpp://",
"drm://",
"drop://",
"dtmi://",
"dtn://",
"dvb://",
"dvx://",
"dweb://",
"ed2k://",
"eid://",
"elsi://",
"embedded://",
"ens://",
"ethereum://",
"example://",
"facetime://",
"fax://",
"feed://",
"feedready://",
"fido://",
"file://",
"filesystem://",
"finger://",
"first-run-pen-experience://",
"fish://",
"fm://",
"ftp://",
"fuchsia-pkg://",
"geo://",
"gg://",
"git://",
"gitoid://",
"gizmoproject://",
"go://",
"gopher://",
"graph://",
"grd://",
"gtalk://",
"h323://",
"ham://",
"hcap://",
"hcp://",
"hs20://",
"http://",
"https://",
"hxxp://",
"hxxps://",
"hydrazone://",
"hyper://",
"iax://",
"icap://",
"icon://",
"im://",
"imap://",
"info://",
"iotdisco://",
"ipfs://",
"ipn://",
"ipns://",
"ipp://",
"ipps://",
"irc://",
"irc6://",
"ircs://",
"iris://",
"iris.beep://",
"iris.lwz://",
"iris.xpc://",
"iris.xpcs://",
"isostore://",
"itms://",
"jabber://",
"jar://",
"jms://",
"keyparc://",
"lastfm://",
"lbry://",
"ldap://",
"ldaps://",
"leaptofrogans://",
"lid://",
"lorawan://",
"lpa://",
"lvlt://",
"machineProvisioningProgressReporter://",
"magnet://",
"mailserver://",
"mailto://",
"maps://",
"market://",
"matrix://",
"message://",
"microsoft.windows.camera://",
"microsoft.windows.camera.multipicker://",
"microsoft.windows.camera.picker://",
"mid://",
"mms://",
"modem://",
"mongodb://",
"moz://",
"ms-access://",
"ms-appinstaller://",
"ms-browser-extension://",
"ms-calculator://",
"ms-drive-to://",
"ms-enrollment://",
"ms-excel://",
"ms-eyecontrolspeech://",
"ms-gamebarservices://",
"ms-gamingoverlay://",
"ms-getoffice://",
"ms-help://",
"ms-infopath://",
"ms-inputapp://",
"ms-launchremotedesktop://",
"ms-lockscreencomponent-config://",
"ms-media-stream-id://",
"ms-meetnow://",
"ms-mixedrealitycapture://",
"ms-mobileplans://",
"ms-newsandinterests://",
"ms-officeapp://",
"ms-people://",
"ms-project://",
"ms-powerpoint://",
"ms-publisher://",
"ms-recall://",
"ms-remotedesktop://",
"ms-remotedesktop-launch://",
"ms-restoretabcompanion://",
"ms-screenclip://",
"ms-screensketch://",
"ms-search://",
"ms-search-repair://",
"ms-secondary-screen-controller://",
"ms-secondary-screen-setup://",
"ms-settings://",
"ms-settings-airplanemode://",
"ms-settings-bluetooth://",
"ms-settings-camera://",
"ms-settings-cellular://",
"ms-settings-cloudstorage://",
"ms-settings-connectabledevices://",
"ms-settings-displays-topology://",
"ms-settings-emailandaccounts://",
"ms-settings-language://",
"ms-settings-location://",
"ms-settings-lock://",
"ms-settings-nfctransactions://",
"ms-settings-notifications://",
"ms-settings-power://",
"ms-settings-privacy://",
"ms-settings-proximity://",
"ms-settings-screenrotation://",
"ms-settings-wifi://",
"ms-settings-workplace://",
"ms-spd://",
"ms-stickers://",
"ms-sttoverlay://",
"ms-transit-to://",
"ms-useractivityset://",
"ms-virtualtouchpad://",
"ms-visio://",
"ms-walk-to://",
"ms-whiteboard://",
"ms-whiteboard-cmd://",
"ms-word://",
"msnim://",
"msrp://",
"msrps://",
"mss://",
"mt://",
"mtqp://",
"mumble://",
"mupdate://",
"mvn://",
"mvrp://",
"mvrps://",
"news://",
"nfs://",
"ni://",
"nih://",
"nntp://",
"notes://",
"num://",
"ocf://",
"oid://",
"onenote://",
"onenote-cmd://",
"opaquelocktoken://",
"openid://",
"openpgp4fpr://",
"otpauth://",
"p1://",
"pack://",
"palm://",
"paparazzi://",
"payment://",
"payto://",
"pkcs11://",
"platform://",
"pop://",
"pres://",
"prospero://",
"proxy://",
"pwid://",
"psyc://",
"pttp://",
"qb://",
"query://",
"quic-transport://",
"redis://",
"rediss://",
"reload://",
"res://",
"resource://",
"rmi://",
"rsync://",
"rtmfp://",
"rtmp://",
"rtsp://",
"rtsps://",
"rtspu://",
"sarif://",
"secondlife://",
"secret-token://",
"service://",
"session://",
"sftp://",
"sgn://",
"shc://",
"shttp://",
"sieve://",
"simpleledger://",
"simplex://",
"sip://",
"sips://",
"skype://",
"smb://",
"smp://",
"sms://",
"smtp://",
"snews://",
"snmp://",
"soap.beep://",
"soap.beeps://",
"soldat://",
"spiffe://",
"spotify://",
"ssb://",
"ssh://",
"starknet://",
"steam://",
"stun://",
"stuns://",
"submit://",
"svn://",
"swh://",
"swid://",
"swidpath://",
"tag://",
"taler://",
"teamspeak://",
"tel://",
"teliaeid://",
"telnet://",
"tftp://",
"things://",
"thismessage://",
"tip://",
"tn3270://",
"tool://",
"turn://",
"turns://",
"tv://",
"udp://",
"unreal://",
"upt://",
"urn://",
"ut2004://",
"uuid-in-package://",
"v-event://",
"vemmi://",
"ventrilo://",
"ves://",
"videotex://",
"vnc://",
"view-source://",
"vscode://",
"vscode-insiders://",
"vsls://",
"w3://",
"wais://",
"web3://",
"wcr://",
"webcal://",
"web+ap://",
"wifi://",
"wpid://",
"ws://",
"wss://",
"wtai://",
"wyciwyg://",
"xcon://",
"xcon-userid://",
"xfire://",
"xmlrpc.beep://",
"xmlrpc.beeps://",
"xmpp://",
"xftp://",
"xrcp://",
"xri://",
"ymsgr://",
"z39.50://",
"z39.50r://",
"z39.50s://"};
static bool is_url(const char *const str, size_t str_len)
{
bool bRet = false;
size_t i;
for (i = 0; i < sizeof(URI_LIST) / sizeof(URI_LIST[0]); i++) {
if (str && (str_len > strlen(URI_LIST[i])) && (0 == strncasecmp(str, URI_LIST[i], strlen(URI_LIST[i])))) {
bRet = true;
goto done;
}
}
done:
return bRet;
}
static void save_urls(cli_ctx *ctx, tag_arguments_t *hrefs, form_data_t *form_data)
{
int i = 0;
json_object *ary = NULL;
if (NULL == hrefs) {
return;
}
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (!(SCAN_STORE_HTML_URIS && SCAN_COLLECT_METADATA && (ctx->this_layer_metadata_json != NULL))) {
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
return;
}
/*Add hrefs*/
for (i = 0; i < hrefs->count; i++) {
if (is_url((const char *)hrefs->value[i], strlen((const char *)hrefs->value[i]))) {
if (NULL == ary) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
ary = cli_jsonarray(ctx->this_layer_metadata_json, HTML_URIS_JSON_KEY);
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
if (!ary) {
libclamav: Add URI scanning support to PDF parser Threat Research requests scanning URIs in PDF files and adding them to the json report file. This change adds URI scanning support to the PDF parser, including support for object references to URIs in PDF files. Jira: CLAM-2588 Fix out-of-order references and other minor improvements. CLAM-2588, CLAM-2757
2025-04-07 16:50:09 -07:00
cli_dbgmsg("[cli_scanhtml] Failed to add \"%s\" entry JSON array\n", HTML_URIS_JSON_KEY);
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
return;
}
}
cli_jsonstr(ary, NULL, (const char *)hrefs->value[i]);
}
}
/*Add form_data*/
for (i = 0; i < (int)form_data->count; i++) {
if (is_url((const char *)form_data->urls[i], strlen((const char *)form_data->urls[i]))) {
if (NULL == ary) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
ary = cli_jsonarray(ctx->this_layer_metadata_json, HTML_URIS_JSON_KEY);
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
if (!ary) {
libclamav: Add URI scanning support to PDF parser Threat Research requests scanning URIs in PDF files and adding them to the json report file. This change adds URI scanning support to the PDF parser, including support for object references to URIs in PDF files. Jira: CLAM-2588 Fix out-of-order references and other minor improvements. CLAM-2588, CLAM-2757
2025-04-07 16:50:09 -07:00
cli_dbgmsg("[cli_scanhtml] Failed to add \"%s\" entry JSON array\n", HTML_URIS_JSON_KEY);
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
return;
}
}
cli_jsonstr(ary, NULL, (const char *)form_data->urls[i]);
}
}
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanhtml(cli_ctx *ctx)
* libclamav/htmlnorm.c: RFC2397 ("data" URL scheme) support. * libclamav/scanner.c: scan RFC2397 data. git-svn: trunk@1244
2005-01-14 14:56:09 +00:00
{
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cl_error_t status = CL_SUCCESS;
char *tempname = NULL;
char fullname[1024];
int fd = -1;
fmap_t *map = ctx->fmap;
uint64_t curr_len = map->len;
* libclamav/htmlnorm.c: RFC2397 ("data" URL scheme) support. * libclamav/scanner.c: scan RFC2397 data. git-svn: trunk@1244
2005-01-14 14:56:09 +00:00
cli_dbgmsg("in cli_scanhtml()\n");
bb #5405: Convert hard-coded maximum file sizes to configurable values with defaults
2012-11-27 17:15:02 -05:00
/* CL_ENGINE_MAX_HTMLNORMALIZE */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (curr_len > ctx->engine->maxhtmlnormalize) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanhtml: exiting (file larger than MaxHTMLNormalize)\n");
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = CL_SUCCESS;
goto done;
don't normalise files larger than 10 MB git-svn: trunk@2623
2007-01-14 13:25:14 +00:00
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (NULL == (tempname = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "html-tmp"))) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = CL_EMEM;
goto done;
}
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(tempname, 0700)) {
don't normalise files larger than 10 MB git-svn: trunk@2623
2007-01-14 13:25:14 +00:00
cli_errmsg("cli_scanhtml: Can't create temporary directory %s\n", tempname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = CL_ETMPDIR;
goto done;
* libclamav/htmlnorm.c: RFC2397 ("data" URL scheme) support. * libclamav/scanner.c: scan RFC2397 data. git-svn: trunk@1244
2005-01-14 14:56:09 +00:00
}
debug message for html tempfiles git-svn: trunk@3723
2008-03-18 22:44:39 +00:00
cli_dbgmsg("cli_scanhtml: using tempdir %s\n", tempname);
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
/* Output JSON Summary Information */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (SCAN_STORE_HTML_URIS && SCAN_COLLECT_METADATA && (ctx->this_layer_metadata_json != NULL)) {
Store URLs from HTML when recording scan metadata json Store URLs found in HTML `<a>` and `<form>` tags during scan of HTML files when recording scan metadata. HTML URL recording will be ON by default, but is a part of the generate-metadata-json feature. The generate-metadata-json feature is OFF by default. This introduces a new general scan option: - libclamav: `CL_SCAN_GENERAL_STORE_HTML_URLS`. - ClamD: `JsonStoreHTMLUrls`. - ClamScan: `--json-store-html-urls` Thank you Matt Jolly for the helpful comment on the pull request.
2024-06-07 10:20:57 -07:00
tag_arguments_t hrefs = {0};
hrefs.scanContents = 1;
form_data_t form_data = {0};
(void)html_normalise_map_form_data(ctx, map, tempname, &hrefs, ctx->dconf, &form_data);
save_urls(ctx, &hrefs, &form_data);
html_tag_arg_free(&hrefs);
html_form_data_tag_free(&form_data);
} else {
(void)html_normalise_map(ctx, map, tempname, NULL, ctx->dconf);
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
snprintf(fullname, 1024, "%s" PATHSEP "nocomment.html", tempname);
fd = open(fullname, O_RDONLY | O_BINARY);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (fd >= 0) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// nocomment.html file exists, so lets scan it.
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
status = cli_scan_desc(fd, ctx, CL_TYPE_HTML, false, NULL, AC_SCAN_VIR, NULL, "no-comment", fullname, LAYER_ATTRIBUTES_NORMALIZED);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
goto done;
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
}
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fd = -1;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/* CL_ENGINE_MAX_HTMLNOTAGS */
curr_len = map->len;
if (curr_len > ctx->engine->maxhtmlnotags) {
/* we're not interested in scanning large files in notags form */
/* TODO: don't even create notags if file is over limit */
cli_dbgmsg("cli_scanhtml: skipping notags (normalized size over MaxHTMLNoTags)\n");
} else {
snprintf(fullname, 1024, "%s" PATHSEP "notags.html", tempname);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
fd = open(fullname, O_RDONLY | O_BINARY);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (fd >= 0) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// notags.html file exists, so lets scan it.
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
status = cli_scan_desc(fd, ctx, CL_TYPE_HTML, false, NULL, AC_SCAN_VIR, NULL, "no-tags", fullname, LAYER_ATTRIBUTES_NORMALIZED);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fd = -1;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
snprintf(fullname, 1024, "%s" PATHSEP "javascript", tempname);
fd = open(fullname, O_RDONLY | O_BINARY);
if (fd >= 0) {
// javascript file exists, so lets scan it (twice, as different types).
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
status = cli_scan_desc(fd, ctx, CL_TYPE_HTML, false, NULL, AC_SCAN_VIR, NULL, "javascript-as-html", fullname, LAYER_ATTRIBUTES_NORMALIZED);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
goto done;
Only create rfc2397 tmp directory when needed HTML normalization creates a tmp directory for storing rfc2397 style links. The vast majority of html does not make use of rfc2397 and thus an excess of empty tmp directories are generated. This commit alters behavior to only create the rfc2397 directory when required if it does not already exist.
2020-03-23 18:53:12 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
status = cli_scan_desc(fd, ctx, CL_TYPE_TEXT_ASCII, false, NULL, AC_SCAN_VIR, NULL, "javascript-as-text-ascii", fullname, LAYER_ATTRIBUTES_NORMALIZED);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
goto done;
}
close(fd);
fd = -1;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
snprintf(fullname, 1024, "%s" PATHSEP "rfc2397", tempname);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = cli_magic_scan_dir(fullname, ctx, LAYER_ATTRIBUTES_NORMALIZED);
if (CL_EOPEN == status) {
/* If the directory doesn't exist, that's fine */
status = CL_SUCCESS;
} else {
goto done;
}
done:
if (fd >= 0) {
close(fd);
}
if (NULL != tempname) {
if (!ctx->engine->keeptmp) {
cli_rmdirs(tempname);
}
free(tempname);
}
return status;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanscript(cli_ctx *ctx)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
{
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cl_error_t ret = CL_SUCCESS;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
const unsigned char *buff;
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
unsigned char *normalized = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct text_norm_state state;
char *tmpname = NULL;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int ofd = -1;
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
struct cli_matcher *target_ac_root;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
uint32_t maxpatlen, offset = 0;
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
struct cli_matcher *generic_ac_root;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct cli_ac_data gmdata, tmdata;
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
int gmdata_initialized = 0;
int tmdata_initialized = 0;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct cli_ac_data *mdata[2];
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cl_fmap_t *new_map = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
fmap_t *map;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
size_t at = 0;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
uint64_t curr_len;
struct cli_target_info info;
if (!ctx || !ctx->engine->root)
return CL_ENULLARG;
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
map = ctx->fmap;
curr_len = map->len;
generic_ac_root = ctx->engine->root[0];
target_ac_root = ctx->engine->root[7];
maxpatlen = target_ac_root ? target_ac_root->maxpatlen : 0;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
PE parsing code improvements, db loading bug fixes Consolidate the PE parsing code into one function. I tried to preserve all existing functionality from the previous, distinct implementations to a large extent (with the exceptions mentioned below). If I noticed potential bugs/improvements, I added a TODO statement about those so that they can be fixed in a smaller commit later. Also, there are more TODOs in places where I'm not entirely sure why certain actions are performed - more research is needed for these. I'm submitting a pull request now so that regression testing can be done, and because merging what I have thus far now will likely have fewer conflicts than if I try to merge later PE parsing code improvements: - PEs without all 16 data directories are parsed more appropriately now - Added lots more debug statements Also: - Allow MAX_BC and MAX_TRACKED_PCRE to be specified via CFLAGS When doing performance testing with the latest CVD, MAX_BC and MAX_TRACKED_PCRE need to be raised to track all the events. Allow these to be specified via CFLAGS by not redefining them if they are already defined - Fix an issue preventing wildcard sizes in .MDB/.MSB rules I'm not sure what the original intent of the check I removed was, but it prevents using wildcard sizes in .MDB/.MSB rules. AFAICT these wildcard sizes should be handled appropriately by the MD5 section hash computation code, so I don't think a check on that is needed. - Fix several issues related to db loading - .imp files will now get loaded if they exist in a directory passed via clamscan's '-d' flag - .pwdb files will now get loaded if they exist in a directory passed via clamscan's '-d' flag even when compiling without yara support - Changes to .imp, .ign, and .ign2 files will now be reflected in calls to cl_statinidir and cl_statchkdir (and also .pwdb files, even when compiling without yara support) - The contents of .sfp files won't be included in some of the signature counts, and the contents of .cud files will be - Any local.gdb files will no longer be loaded twice - For .imp files, you are no longer required to specify a minimum flevel for wildcard rules, since this isn't needed
2019-01-08 00:09:08 -05:00
// Initialize info so it's safe to pass to destroy later
cli_targetinfo_init(&info);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("in cli_scanscript()\n");
/* CL_ENGINE_MAX_SCRIPTNORMALIZE */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (curr_len > ctx->engine->maxscriptnormalize) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanscript: exiting (file larger than MaxScriptSize)\n");
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
ret = CL_SUCCESS;
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Remove additional memory allocation limits relating to signature load Variables like the number of signature parts are considered trusted user input and so allocations based on those values need not check the memory allocation limit. Specifically for the allocation of the normalized buffer in cli_scanscript, I determined that the size of SCANBUFF is fixed and so safe, and the maxpatlen comes from the signature load and is therefore also trusted, so we do not need to check the allocation limit.
2024-01-09 17:17:48 -05:00
if (!(normalized = malloc(SCANBUFF + maxpatlen))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanscript: Unable to malloc %u bytes\n", SCANBUFF);
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
ret = CL_EMEM;
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
text_normalize_init(&state, normalized, SCANBUFF + maxpatlen);
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
if ((ret = cli_ac_initdata(&tmdata, target_ac_root ? target_ac_root->ac_partsigs : 0, target_ac_root ? target_ac_root->ac_lsigs : 0, target_ac_root ? target_ac_root->ac_reloff_num : 0, CLI_DEFAULT_AC_TRACKLEN))) {
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
tmdata_initialized = 1;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
if ((ret = cli_ac_initdata(&gmdata, generic_ac_root->ac_partsigs, generic_ac_root->ac_lsigs, generic_ac_root->ac_reloff_num, CLI_DEFAULT_AC_TRACKLEN))) {
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
gmdata_initialized = 1;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* dump to disk only if explicitly asked to
* or if necessary to check relative offsets,
* otherwise we can process just in-memory */
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
if (ctx->engine->keeptmp || (target_ac_root && (target_ac_root->ac_reloff_num > 0 || target_ac_root->linked_bcs))) {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_gentempfd(ctx->this_layer_tmpdir, &tmpname, &ofd))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanscript: Can't generate temporary file/descriptor\n");
goto done;
}
if (ctx->engine->keeptmp)
cli_dbgmsg("cli_scanscript: saving normalized file to %s\n", tmpname);
}
mdata[0] = &tmdata;
mdata[1] = &gmdata;
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
/* If there's a relative offset in target_ac_root or triggered bytecodes, normalize to file.*/
if (target_ac_root && (target_ac_root->ac_reloff_num > 0 || target_ac_root->linked_bcs)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
size_t map_off = 0;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while (map_off < map->len) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
size_t written;
if (!(written = text_normalize_map(&state, map, map_off)))
break;
map_off += written;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (write(ofd, state.out, state.out_pos) == -1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanscript: can't write to file %s\n", tmpname);
ret = CL_EWRITE;
goto done;
}
text_normalize_reset(&state);
}
/* Temporarily store the normalized file map in the context. */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
new_map = fmap_new(ofd, 0, 0, NULL, tmpname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (new_map == NULL) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanscript: could not map file %s\n", tmpname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
goto done;
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
/* Perform cli_scan_fmap with child fmap */
ret = cli_recursion_stack_push(ctx, new_map, CL_TYPE_TEXT_ASCII, true, LAYER_ATTRIBUTES_NORMALIZED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (CL_SUCCESS != ret) {
cli_dbgmsg("Failed to scan fmap.\n");
goto done;
}
/* scan map */
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_scan_fmap(ctx, CL_TYPE_TEXT_ASCII, false, NULL, AC_SCAN_VIR, NULL);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
(void)cli_recursion_stack_pop(ctx); /* Restore the parent fmap */
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
goto done;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* Since the above is moderately costly all in all,
* do the old stuff if there's no relative offsets. */
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
if (target_ac_root) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_targetinfo(&info, 7, ctx);
Matcher: Remove allmatch checks and significantly tidy code Significantly tidy the `cli_scan_fmap()` function, and add comments to explain how it all works. Add SHA1 and SHA256 digest variables to the FMAP structure in addition to the existing MD5. Add a function to set the hash so that when we calculate the hashes for hash matching, we save them for subsequent FP matching. This enabled me to remove the extra "hash-only" FP check from `cli_scan_fmap()`. This will also make it easier to switch the clean cache hash algorithm to SHA256 in the future. Remove extra allmatch checks that are no longer required. Add a new header to prevent #include order issues.
2022-08-18 20:00:33 -07:00
ret = cli_ac_caloff(target_ac_root, &tmdata, &info);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (ret)
goto done;
}
* libclamav/htmlnorm.c: RFC2397 ("data" URL scheme) support. * libclamav/scanner.c: scan RFC2397 data. git-svn: trunk@1244
2005-01-14 14:56:09 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while (1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
size_t len = MIN(map->pgsz, map->len - at);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
buff = fmap_need_off_once(map, at, len);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
at += len;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!buff || !len || state.out_pos + len > state.out_len) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* flush if error/EOF, or too little buffer space left */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if ((ofd != -1) && (write(ofd, state.out, state.out_pos) == -1)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanscript: can't write to file %s\n", tmpname);
close(ofd);
ofd = -1;
/* we can continue to scan in memory */
}
/* when we flush the buffer also scan */
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = cli_scan_buff(state.out, state.out_pos, offset, ctx, CL_TYPE_TEXT_ASCII, mdata);
if (CL_SUCCESS != ret) {
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (ctx->scanned)
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
*ctx->scanned += state.out_pos;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
offset += state.out_pos;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* carry over maxpatlen from previous buffer */
if (state.out_pos > maxpatlen)
memmove(state.out, state.out + state.out_pos - maxpatlen, maxpatlen);
text_normalize_reset(&state);
state.out_pos = maxpatlen;
}
if (!len)
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!buff || text_normalize_buffer(&state, buff, len) != len) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanscript: short read during normalizing\n");
bb #5405: Convert hard-coded maximum file sizes to configurable values with defaults
2012-11-27 17:15:02 -05:00
}
}
* libclamav/htmlnorm.c: RFC2397 ("data" URL scheme) support. * libclamav/scanner.c: scan RFC2397 data. git-svn: trunk@1244
2005-01-14 14:56:09 +00:00
}
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_exp_eval(ctx, target_ac_root, &tmdata, NULL);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
goto done;
}
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_exp_eval(ctx, generic_ac_root, &gmdata, NULL);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
goto done;
enable JS normalizer git-svn: trunk@3919
2008-07-08 11:33:32 +00:00
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
done:
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (NULL != new_map) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(new_map);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
PE parsing code improvements, db loading bug fixes Consolidate the PE parsing code into one function. I tried to preserve all existing functionality from the previous, distinct implementations to a large extent (with the exceptions mentioned below). If I noticed potential bugs/improvements, I added a TODO statement about those so that they can be fixed in a smaller commit later. Also, there are more TODOs in places where I'm not entirely sure why certain actions are performed - more research is needed for these. I'm submitting a pull request now so that regression testing can be done, and because merging what I have thus far now will likely have fewer conflicts than if I try to merge later PE parsing code improvements: - PEs without all 16 data directories are parsed more appropriately now - Added lots more debug statements Also: - Allow MAX_BC and MAX_TRACKED_PCRE to be specified via CFLAGS When doing performance testing with the latest CVD, MAX_BC and MAX_TRACKED_PCRE need to be raised to track all the events. Allow these to be specified via CFLAGS by not redefining them if they are already defined - Fix an issue preventing wildcard sizes in .MDB/.MSB rules I'm not sure what the original intent of the check I removed was, but it prevents using wildcard sizes in .MDB/.MSB rules. AFAICT these wildcard sizes should be handled appropriately by the MD5 section hash computation code, so I don't think a check on that is needed. - Fix several issues related to db loading - .imp files will now get loaded if they exist in a directory passed via clamscan's '-d' flag - .pwdb files will now get loaded if they exist in a directory passed via clamscan's '-d' flag even when compiling without yara support - Changes to .imp, .ign, and .ign2 files will now be reflected in calls to cl_statinidir and cl_statchkdir (and also .pwdb files, even when compiling without yara support) - The contents of .sfp files won't be included in some of the signature counts, and the contents of .cud files will be - Any local.gdb files will no longer be loaded twice - For .imp files, you are no longer required to specify a minimum flevel for wildcard rules, since this isn't needed
2019-01-08 00:09:08 -05:00
cli_targetinfo_destroy(&info);
Address code-review comments, fix several memleaks Changes include: - Fixing several memory leaks noticed when running with ASan - Adds documentation for several functions and structs - Simplifies the interface for using cli_targetinfo_init/destroy and cli_exe_info_init/destroy - A few other minor changes
2019-03-12 12:45:19 -04:00
if (NULL != normalized) {
free(normalized);
}
if (tmdata_initialized) {
cli_ac_freedata(&tmdata);
}
if (gmdata_initialized) {
cli_ac_freedata(&gmdata);
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (ofd != -1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(ofd);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (tmpname != NULL) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!ctx->engine->keeptmp) {
(void)cli_unlink(tmpname);
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
* libclamav/htmlnorm.c: RFC2397 ("data" URL scheme) support. * libclamav/scanner.c: scan RFC2397 data. git-svn: trunk@1244
2005-01-14 14:56:09 +00:00
}
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanhtml_utf16(cli_ctx *ctx)
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
{
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cl_error_t status = CL_ERROR;
char *tempname = NULL;
char *decoded = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
const char *buff;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
int fd = -1;
int bytes;
size_t at = 0;
fmap_t *new_map = NULL;
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
cli_dbgmsg("in cli_scanhtml_utf16()\n");
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(tempname = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "html-utf16-tmp"))) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EMEM;
goto done;
}
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
Change permission for new tmp files from RWX to RW
2020-03-30 20:42:44 -04:00
if ((fd = open(tempname, O_RDWR | O_CREAT | O_TRUNC | O_BINARY, S_IRUSR | S_IWUSR)) < 0) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanhtml_utf16: Can't create file %s\n", tempname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EOPEN;
goto done;
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
}
debug message for html tempfiles git-svn: trunk@3723
2008-03-18 22:44:39 +00:00
cli_dbgmsg("cli_scanhtml_utf16: using tempfile %s\n", tempname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
while (at < ctx->fmap->len) {
bytes = MIN(ctx->fmap->len - at, ctx->fmap->pgsz * 16);
if (!(buff = fmap_need_off_once(ctx->fmap, at, bytes))) {
status = CL_EREAD;
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
at += bytes;
decoded = cli_utf16toascii(buff, bytes);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (decoded) {
if (write(fd, decoded, bytes / 2) == -1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanhtml_utf16: Can't write to file %s\n", tempname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EWRITE;
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
free(decoded);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
decoded = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
}
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
new_map = fmap_new(fd, 0, 0, NULL, tempname);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (NULL == new_map) {
cli_errmsg("cli_scanhtml_utf16: failed to create fmap for ascii HTML file decoded from utf16: %s\n.", tempname);
status = CL_EMEM;
goto done;
}
(some) html to fmap
2009-08-31 06:16:12 +02:00
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
/* Perform exp_eval with child fmap */
status = cli_recursion_stack_push(ctx, new_map, CL_TYPE_HTML, true, LAYER_ATTRIBUTES_NORMALIZED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (CL_SUCCESS != status) {
cli_dbgmsg("Failed to scan fmap.\n");
goto done;
}
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = cli_scanhtml(ctx);
(void)cli_recursion_stack_pop(ctx); /* Restore the parent fmap */
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
goto done;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
done:
if (NULL != new_map) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(new_map);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
if (-1 != fd) {
close(fd);
}
if (NULL != decoded) {
free(decoded);
}
if (NULL != tempname) {
if (!ctx->engine->keeptmp) {
(void)cli_unlink(tempname);
} else {
cli_dbgmsg("cli_scanhtml_utf16: Decoded HTML data saved in %s\n", tempname);
}
free(tempname);
}
return status;
add support for UTF16 encoded HTML files git-svn: trunk@2430
2006-10-25 15:40:47 +00:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
static cl_error_t cli_ole2_scan_tempdir(
cli_ctx *ctx,
const char *dir,
struct uniq *files,
int has_vba,
int has_xlm,
int has_image)
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
DIR *dd = NULL;
int has_macros = 0;
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
struct dirent *dent;
STATBUF statbuf;
char *subdirectory = NULL;
more limits and ole2 collision fixup git-svn-id: file:///var/lib/svn/clamav-devel/branches/newlimits@3608 77e5149b-7576-45b1-b177-96237e5ba77b
2008-02-11 13:18:41 +00:00
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
cli_dbgmsg("cli_ole2_scan_tempdir: %s\n", dir);
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
/* Output JSON Summary Information */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (SCAN_COLLECT_METADATA && (ctx->this_layer_metadata_json != NULL)) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
(void)cli_ole2_tempdir_scan_summary(dir, ctx, files);
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
status = cli_ole2_tempdir_scan_embedded_ole10(dir, ctx, files);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (has_vba) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = cli_ole2_tempdir_scan_vba(dir, ctx, files, &has_macros);
if (CL_SUCCESS != status) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
Modernize VBA code extraction from Microsoft Office files - Existing VBA extraction code uses undocumented cache structures. This code uses the documented way of accessing VBA projects. - Adds additional detail to the dumped information: Project name, Project doc string, ... All VBA projects are dumped into a single file. - Malware authors are currently evading detection by spreading malicious code over several projects. It is hard to write signatures if only part of the malicious code is visible.
2020-04-28 13:32:07 -07:00
}
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
status = cli_ole2_tempdir_scan_vba_new(dir, ctx, files, &has_macros);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
}
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
}
Fix XLM macro false positives in XLS docs containing images The XLS parsing code for extracting images and XLM macros is alerting with Heuristics.OLE2.ContainsMacros.XLM if any images are found if the --alert-macros (AlertOLE2Macros) option is enabled. This fix moves the alert logic before the XLM & image extraction logic where we know at least one macro exists, but before we try to extract it I've wrapped it with an "if (has_xlm)" condition. Resolves: - https://github.com/Cisco-Talos/clamav/issues/423 - https://bugzilla.clamav.net/show_bug.cgi?id=12844
2022-01-25 21:48:27 -08:00
if (has_xlm) {
if (SCAN_HEURISTIC_MACROS) {
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
status = cli_append_potentially_unwanted(ctx, "Heuristics.OLE2.ContainsMacros.XLM");
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
goto done;
Fix XLM macro false positives in XLS docs containing images The XLS parsing code for extracting images and XLM macros is alerting with Heuristics.OLE2.ContainsMacros.XLM if any images are found if the --alert-macros (AlertOLE2Macros) option is enabled. This fix moves the alert logic before the XLM & image extraction logic where we know at least one macro exists, but before we try to extract it I've wrapped it with an "if (has_xlm)" condition. Resolves: - https://github.com/Cisco-Talos/clamav/issues/423 - https://bugzilla.clamav.net/show_bug.cgi?id=12844
2022-01-25 21:48:27 -08:00
}
}
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (has_xlm || has_image) {
/* TODO: Consider moving image extraction to handler_enum and
* removing the has_image and found_image stuff. */
Fix XLM macro false positives in XLS docs containing images The XLS parsing code for extracting images and XLM macros is alerting with Heuristics.OLE2.ContainsMacros.XLM if any images are found if the --alert-macros (AlertOLE2Macros) option is enabled. This fix moves the alert logic before the XLM & image extraction logic where we know at least one macro exists, but before we try to extract it I've wrapped it with an "if (has_xlm)" condition. Resolves: - https://github.com/Cisco-Talos/clamav/issues/423 - https://bugzilla.clamav.net/show_bug.cgi?id=12844
2022-01-25 21:48:27 -08:00
status = cli_ole2_tempdir_scan_for_xlm_and_images(dir, ctx, files);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
}
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (has_xlm || has_vba) {
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_magic_scan_dir(dir, ctx, LAYER_ATTRIBUTES_NONE);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
Heuristic macro detection for imp VBA extraction Notably the commit adds a heuristic alert when VBA is extracted using the new VBA extraction code and similarly adds "HasMacros":true to the JSON scan properties. In addition, a change was added to the cli_sanitize_filepath() function so it converts posix pathseps to Windows pathseps on Windows and also outputs a sanitized basename pointer (optional) which is used when generating a temporary filename so that using a prefix with pathseps in it won't cause file creation failures (observed with --leave-temps where original filenames are incorporated into temporarily filenames). Included soem error handling improvements for cli_vba_scandir() to better track alert and macro detections. Downgraded utf8 conversion error messages to debug messages because they are too verbose in files with invalid filenames (observed in some malware). Changed the xlm macro and vba project temp filenames to include "xlm_macros" and "vba_project" prefix, to make it easier to find them. Relocated XLM and VBA temp files from the top-level tmp directory to the current sub_tmpdir, so tempfiles for a given scan are more organized.
2020-08-07 23:48:20 -07:00
}
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
}
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
/* ACAB: since we now hash filenames and handle collisions we
* could avoid recursion by removing the block below and by
* flattening the paths in ole2_walk_property_tree (case 1) */
if ((dd = opendir(dir)) != NULL) {
while ((dent = readdir(dd))) {
if (dent->d_ino) {
if (strcmp(dent->d_name, ".") && strcmp(dent->d_name, "..")) {
/* build the full name */
Rename clamav memory allocation functions We have some special functions to wrap malloc, calloc, and realloc to make sure we don't allocate more than some limit, similar to the max-filesize and max-scansize limits. Our wrappers are really only needed when allocating memory for scans based on untrusted user input, where a scan file could have bytes that claim you need to allocate some ridiculous amount of memory. Right now they're named: - cli_malloc - cli_calloc - cli_realloc - cli_realloc2 ... and these names do not convey their purpose This commit renames them to: - cli_max_malloc - cli_max_calloc - cli_max_realloc - cli_max_realloc2 The realloc ones also have an additional feature in that they will not free your pointer if you try to realloc to 0 bytes. Freeing the memory is undefined by the C spec, and only done with some realloc implementations, so this stabilizes on the behavior of not doing that, which should prevent accidental double-free's. So for the case where you may want to realloc and do not need to have a maximum, this commit adds the following functions: - cli_safer_realloc - cli_safer_realloc2 These are used for the MPOOL_REALLOC and MPOOL_REALLOC2 macros when MPOOL is disabled (e.g. because mmap-support is not found), so as to match the behavior in the mpool_realloc/2 functions that do not make use of the allocation-limit.
2022-05-09 14:28:34 -07:00
subdirectory = malloc(strlen(dir) + strlen(dent->d_name) + 2);
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (!subdirectory) {
cli_dbgmsg("cli_ole2_tempdir_scan_vba: Unable to allocate memory for subdirectory path\n");
status = CL_EMEM;
break;
}
sprintf(subdirectory, "%s" PATHSEP "%s", dir, dent->d_name);
/* stat the file */
if (LSTAT(subdirectory, &statbuf) != -1) {
if (S_ISDIR(statbuf.st_mode) && !S_ISLNK(statbuf.st_mode)) {
/*
* Process subdirectory
*/
status = cli_ole2_scan_tempdir(
ctx,
subdirectory,
files,
has_vba,
has_xlm,
has_image);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != status) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
}
}
}
free(subdirectory);
subdirectory = NULL;
}
}
}
} else {
cli_dbgmsg("VBADir: Can't open directory %s.\n", dir);
status = CL_EOPEN;
goto done;
}
done:
if (NULL != dd) {
closedir(dd);
}
if (NULL != subdirectory) {
free(subdirectory);
}
return status;
}
static cl_error_t cli_scanole2(cli_ctx *ctx)
{
char *dir = NULL;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
struct uniq *files = NULL;
int has_vba = 0;
int has_xlm = 0;
int has_image = 0;
cli_dbgmsg("in cli_scanole2()\n");
/* generate the temporary directory */
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (NULL == (dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "ole2-tmp"))) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
ret = CL_EMEM;
goto done;
}
if (mkdir(dir, 0700)) {
cli_dbgmsg("OLE2: Can't create temporary directory %s\n", dir);
free(dir);
dir = NULL;
ret = CL_ETMPDIR;
goto done;
}
ret = cli_ole2_extract(dir, ctx, &files, &has_vba, &has_xlm, &has_image);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
goto done;
}
if (files) {
/*
* Files containing the document summary, any VBA or XLM macros, or
* images were previously extracted from an ole2 file.
* This happens if cli_ole2_extract() executes the handler_writer()
* because XLM, VBA, or images were found.
* So now we need to process them.
*
* TODO: consider maybe processes all that stuff in memory instead of
* writing everything to temp files?
*/
ret = cli_ole2_scan_tempdir(
ctx,
dir,
files,
has_vba,
has_xlm,
has_image);
}
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
done:
XLM (Excel 4.0) macro detection and extraction XLM is a macro language in Excel that was used before VBA (before 1996). It is still parsed and executed by modern Excel and is gaining popularity with malware authors. This patch adds rudimentary support for detecting and extracting Excel 4.0 (XLM) macros. The code is based on Didier Steven's plugin_biff for oletools.py.
2020-04-29 14:19:41 -07:00
if (files) {
uniq_free(files);
}
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
if (NULL != dir) {
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
if (!ctx->engine->keeptmp) {
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
cli_rmdirs(dir);
OLE2: doc summary, emb. ole10 scans skipped if XLM macro found Also resolved the following issue: If XLM (and now images) are found when parsing an ole2 files the following other embedded content may not be processed: - document summary metadata - embedded ole10 files - ole2 temp subdirectories (i.e. recursion) The logic to process the above ole2 extracted temp files was present in the function which processes extracted VBA. When we added support for extracting XLM macros, processing these other data was lost. Really, the above need to be processed if any temp files were saved. I fixed this by restructuring the features to extract any type of temp file into separate functions per type of temp file. I then wrappped those in an ole2 temp dir scanning function. OLE2 temp directory scanning is recursive if there are subdirectories.
2021-07-03 12:06:52 -07:00
}
Fix newly introduced VBA/XLM OLE2 bugs Fix an infinite loop in the new XLM macro parser. Fix error handling, resource cleanup in OLE2 parser. Fix issues tracking detected "viruses" in VBA & OLE2 parsers affecting non-allmatch (regular) scan mode, wherein multiple viruses may be found but each record lost and the overall detection comes up clean. Also silence switch() fall-through warning for WORD/PPT/XL/HWP (OOXML) file type fall-throughs to the ZIP parser (because they are zips). Also silence switch() fall-through warning when handling the limits- exceeded error types, checking for the limits-exceeded heuristic, and continuing on to bail out with a clean verdict.
2020-08-06 22:15:33 -07:00
free(dir);
}
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
return ret;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scantar(cli_ctx *ctx, unsigned int posix)
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
{
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
char *dir;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
cli_dbgmsg("in cli_scantar()\n");
/* generate temporary directory */
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "tar-tmp")))
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return CL_EMEM;
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(dir, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("Tar: Can't create temporary directory %s\n", dir);
free(dir);
return CL_ETMPDIR;
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
}
fmapify tar
2011-06-13 11:57:59 +03:00
ret = cli_untar(dir, posix, ctx);
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
cli_rmdirs(dir);
improve zip and tar support, minor cleanups git-svn: trunk@836
2004-09-07 21:19:02 +00:00
free(dir);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanscrenc(cli_ctx *ctx)
re-write HTML code git-svn: trunk@849
2004-09-13 10:30:14 +00:00
{
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
char *tempname;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
re-write HTML code git-svn: trunk@849
2004-09-13 10:30:14 +00:00
cli_dbgmsg("in cli_scanscrenc()\n");
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(tempname = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "screnc-tmp")))
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return CL_EMEM;
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(tempname, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("CHM: Can't create temporary directory %s\n", tempname);
free(tempname);
return CL_ETMPDIR;
re-write HTML code git-svn: trunk@849
2004-09-13 10:30:14 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (html_screnc_decode(ctx->fmap, tempname))
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_dir(tempname, ctx, LAYER_ATTRIBUTES_NONE);
re-write HTML code git-svn: trunk@849
2004-09-13 10:30:14 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
cli_rmdirs(tempname);
re-write HTML code git-svn: trunk@849
2004-09-13 10:30:14 +00:00
free(tempname);
return ret;
}
cleanup git-svn: trunk@894
2004-09-18 00:14:00 +00:00
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanriff(cli_ctx *ctx)
activate RIFF code git-svn: trunk@1311
2005-02-05 15:50:18 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
activate RIFF code git-svn: trunk@1311
2005-02-05 15:50:18 +00:00
bb11805 fix multiple results. Refactor false positive and heuristic precedence logic.
2017-04-18 12:03:36 -04:00
if (cli_check_riff_exploit(ctx) == 2)
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
ret = cli_append_potentially_unwanted(ctx, "Heuristics.Exploit.W32.MS05-002");
activate RIFF code git-svn: trunk@1311
2005-02-05 15:50:18 +00:00
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scancryptff(cli_ctx *ctx)
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS, ndesc;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
unsigned int i;
const unsigned char *src;
unsigned char *dest = NULL;
char *tempfile;
size_t pos;
size_t bread;
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
/* Skip the CryptFF file header */
fmapify cryptff
2011-06-13 12:12:01 +03:00
pos = 0x10;
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
Optimization: replace limited allocation calls There are a large number of allocations for fix sized buffers using the `cli_malloc` and `cli_calloc` calls that check if the requested size is larger than our allocation threshold for allocations based on untrusted input. These allocations will *always* be higher than the threshold, so the extra stack frame and check for these calls is a waste of CPU. This commit replaces needless calls with A -> B: - cli_malloc -> malloc - cli_calloc -> calloc - CLI_MALLOC -> MALLOC - CLI_CALLOC -> CALLOC I also noticed that our MPOOL_MALLOC / MPOOL_CALLOC are not limited by the max-allocation threshold, when MMAP is found/enabled. But the alternative was set to cli_malloc / cli_calloc when disabled. I changed those as well. I didn't change the cli_realloc/2 calls because our version of realloc not only implements a threshold but also stabilizes the undefined behavior in realloc to protect against accidental double-free's. It may be worth implementing a cli_realloc that doesn't have the threshold built-in, however, so as to allow reallocaitons for things like buffers for loading signatures, which aren't subject to the same concern as allocations for scanning possible malware. There was one case in mbox.c where I changed MALLOC -> CLI_MALLOC, because it appears to be allocating based on untrusted input.
2022-05-08 14:59:09 -07:00
if ((dest = (unsigned char *)malloc(FILEBUFF)) == NULL) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("CryptFF: Can't allocate memory\n");
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
return CL_EMEM;
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (!(tempfile = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "cryptff"))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(dest);
return CL_EMEM;
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
}
Change permission for new tmp files from RWX to RW
2020-03-30 20:42:44 -04:00
if ((ndesc = open(tempfile, O_RDWR | O_CREAT | O_TRUNC | O_BINARY, S_IRUSR | S_IWUSR)) < 0) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("CryptFF: Can't create file %s\n", tempfile);
free(dest);
free(tempfile);
return CL_ECREAT;
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
for (; (src = fmap_need_off_once_len(ctx->fmap, pos, FILEBUFF, &bread)) && bread; pos += bread) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
for (i = 0; i < bread; i++)
dest[i] = src[i] ^ (unsigned char)0xff;
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
if (cli_writen(ndesc, dest, bread) == (size_t)-1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("CryptFF: Can't write to descriptor %d\n", ndesc);
free(dest);
close(ndesc);
free(tempfile);
return CL_EWRITE;
}
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
}
free(dest);
cli_dbgmsg("CryptFF: Scanning decrypted data\n");
Fix issue reporting all Heuristic/PUA matches in allmatch mode When potentially unwanted indicators are found, they are not reported right away, except in heuristic-precedence mode. At the end of the scan, we then report all of the heuristic/PUA alerts. In my initial overhaul of the allmatch mode, I introduced an issue where it is never reporting more than 1 heuristic/PUA alert. This commit fixes that by reporting all potentially-unwanted indicators at the end. Note that instead of using `cli_virus_found_cb()` which would report the top layer file descriptor as well in the callback, I'm reporting `-1` for the file descriptor instead, because we don't know when the alert was added. If it was at a deeper layer in the file, we would not longer have access to it at the end of the scan. Also removed excessive debug statements '... infected with ...' that use `cli_get_last_virus()` because it doesn't really reflect all of the matches in allmatch mode, because the match would've been reported previously (so it is excessive), and because I don't like the word 'infected'. :)
2022-08-16 19:13:05 -07:00
ret = cli_magic_scan_desc(ndesc, tempfile, ctx, NULL, LAYER_ATTRIBUTES_NONE);
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
close(ndesc);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (ctx->engine->keeptmp) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("CryptFF: Decompressed data saved in %s\n", tempfile);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
} else {
if (CL_SUCCESS != cli_unlink(tempfile)) {
ret = CL_EUNLINK;
}
}
add support for CryptFF git-svn: trunk@1756
2005-11-14 21:02:26 +00:00
free(tempfile);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanpdf(cli_ctx *ctx, off_t offset)
activate PDF code git-svn: trunk@1510
2005-05-03 00:10:46 +00:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t ret;
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
char *dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "pdf-tmp");
activate PDF code git-svn: trunk@1510
2005-05-03 00:10:46 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!dir)
return CL_EMEM;
activate PDF code git-svn: trunk@1510
2005-05-03 00:10:46 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(dir, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Can't create temporary directory for PDF file %s\n", dir);
free(dir);
return CL_ETMPDIR;
activate PDF code git-svn: trunk@1510
2005-05-03 00:10:46 +00:00
}
binhex, pdf, unzip_single to fmap
2009-08-31 05:37:43 +02:00
ret = cli_pdf(dir, ctx, offset);
activate PDF code git-svn: trunk@1510
2005-05-03 00:10:46 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
cli_rmdirs(dir);
activate PDF code git-svn: trunk@1510
2005-05-03 00:10:46 +00:00
free(dir);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scantnef(cli_ctx *ctx)
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t ret;
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
char *dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "tnef-tmp");
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!dir)
return CL_EMEM;
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(dir, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Can't create temporary directory for tnef file %s\n", dir);
free(dir);
return CL_ETMPDIR;
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
}
fmapify tnef
2011-06-13 11:47:41 +03:00
ret = cli_tnef(dir, ctx);
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (ret == CL_SUCCESS)
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_dir(dir, ctx, LAYER_ATTRIBUTES_NONE);
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
cli_rmdirs(dir);
update git-svn: trunk@1426
2005-03-25 15:17:13 +00:00
free(dir);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanuuencoded(cli_ctx *ctx)
Decode uuencode files git-svn: trunk@1822
2006-01-21 18:37:48 +00:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t ret;
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
char *dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "uuencoded-tmp");
Decode uuencode files git-svn: trunk@1822
2006-01-21 18:37:48 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!dir)
return CL_EMEM;
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(dir, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Can't create temporary directory for uuencoded file %s\n", dir);
free(dir);
return CL_ETMPDIR;
Decode uuencode files git-svn: trunk@1822
2006-01-21 18:37:48 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
ret = cli_uuencode(dir, ctx->fmap);
Decode uuencode files git-svn: trunk@1822
2006-01-21 18:37:48 +00:00
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (ret == CL_SUCCESS)
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_dir(dir, ctx, LAYER_ATTRIBUTES_NONE);
Decode uuencode files git-svn: trunk@1822
2006-01-21 18:37:48 +00:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!ctx->engine->keeptmp)
cli_rmdirs(dir);
Decode uuencode files git-svn: trunk@1822
2006-01-21 18:37:48 +00:00
free(dir);
return ret;
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scanmail(cli_ctx *ctx)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
char *dir = NULL;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t ret;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_dbgmsg("Starting cli_scanmail()\n");
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
improve generating of temporary file names git-svn: trunk@1101
2004-11-16 17:10:47 +00:00
/* generate the temporary directory */
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (NULL == (dir = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "mail-tmp"))) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = CL_EMEM;
goto done;
}
always check return value of cli_gentemp() git-svn: trunk@3696
2008-03-06 20:19:22 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (mkdir(dir, 0700)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("Mail: Can't create temporary directory %s\n", dir);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = CL_ETMPDIR;
goto done;
improve generating of temporary file names git-svn: trunk@1101
2004-11-16 17:10:47 +00:00
}
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
improve generating of temporary file names git-svn: trunk@1101
2004-11-16 17:10:47 +00:00
/*
* Extract the attachments into the temporary directory
*/
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
ret = cli_mbox(dir, ctx);
if (CL_SUCCESS != ret) {
goto done;
improve generating of temporary file names git-svn: trunk@1101
2004-11-16 17:10:47 +00:00
}
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_dir(dir, ctx, LAYER_ATTRIBUTES_NONE);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_SUCCESS != ret) {
goto done;
}
improve generating of temporary file names git-svn: trunk@1101
2004-11-16 17:10:47 +00:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
done:
if (NULL != dir) {
if (!ctx->engine->keeptmp) {
cli_rmdirs(dir);
}
free(dir);
}
improve generating of temporary file names git-svn: trunk@1101
2004-11-16 17:10:47 +00:00
return ret;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
static cl_error_t cli_scan_structured(cli_ctx *ctx)
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
{
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
char buf[8192];
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
size_t result = 0;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
unsigned int cc_count = 0;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
unsigned int ssn_count = 0;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
bool done = false;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
fmap_t *map;
size_t pos = 0;
Add DLP feature to detect credit cards only Add Data-Loss-Prevention option to detect credit cards only, excluding debit and private label cards where possible. You can select the credit card-only DLP mode for clamscan with the `--structured-cc-mode` command-line option. You can select the credit card-only DLP mode for clamd with the `StructuredCCOnly` clamd.conf config option. This patch also adds credit card matching for additional vendors: - Mastercard 2016 - China Union Pay - Discover 2009
2016-04-21 12:13:57 -04:00
int (*ccfunc)(const unsigned char *buffer, size_t length, int cc_only);
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
int (*ssnfunc)(const unsigned char *buffer, size_t length);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (ctx == NULL)
return CL_ENULLARG;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
map = ctx->fmap;
BB#5475
2012-07-12 10:21:00 -04:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (ctx->engine->min_cc_count == 1)
ccfunc = dlp_has_cc;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
else
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
ccfunc = dlp_get_cc_count;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
switch (SCAN_HEURISTIC_STRUCTURED_SSN_NORMAL | SCAN_HEURISTIC_STRUCTURED_SSN_STRIPPED) {
case (CL_SCAN_HEURISTIC_STRUCTURED_SSN_NORMAL | CL_SCAN_HEURISTIC_STRUCTURED_SSN_STRIPPED):
if (ctx->engine->min_ssn_count == 1)
ssnfunc = dlp_has_ssn;
else
ssnfunc = dlp_get_ssn_count;
break;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_SCAN_HEURISTIC_STRUCTURED_SSN_NORMAL:
if (ctx->engine->min_ssn_count == 1)
ssnfunc = dlp_has_normal_ssn;
else
ssnfunc = dlp_get_normal_ssn_count;
break;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_SCAN_HEURISTIC_STRUCTURED_SSN_STRIPPED:
if (ctx->engine->min_ssn_count == 1)
ssnfunc = dlp_has_stripped_ssn;
else
ssnfunc = dlp_get_stripped_ssn_count;
break;
DLP updates git-svn: trunk@3798
2008-04-18 17:14:20 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
default:
ssnfunc = NULL;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
}
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
while (!done && ((result = fmap_readn(map, buf, pos, 8191)) > 0) && (result != (size_t)-1)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
pos += result;
Add DLP feature to detect credit cards only Add Data-Loss-Prevention option to detect credit cards only, excluding debit and private label cards where possible. You can select the credit card-only DLP mode for clamscan with the `--structured-cc-mode` command-line option. You can select the credit card-only DLP mode for clamd with the `StructuredCCOnly` clamd.conf config option. This patch also adds credit card matching for additional vendors: - Mastercard 2016 - China Union Pay - Discover 2009
2016-04-21 12:13:57 -04:00
if ((cc_count += ccfunc((const unsigned char *)buf, result,
dlp - clang-format'd
2019-08-27 17:33:22 -04:00
(ctx->options->heuristic & CL_SCAN_HEURISTIC_STRUCTURED_CC) ? 1 : 0)) >= ctx->engine->min_cc_count) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
done = true;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
DLP updates git-svn: trunk@3798
2008-04-18 17:14:20 +00:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ssnfunc && ((ssn_count += ssnfunc((const unsigned char *)buf, result)) >= ctx->engine->min_ssn_count)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
done = true;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cc_count != 0 && cc_count >= ctx->engine->min_cc_count) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scan_structured: %u credit card numbers detected\n", cc_count);
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
if (CL_VIRUS == cli_append_potentially_unwanted(ctx, "Heuristics.Structured.CreditCardNumber")) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return CL_VIRUS;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ssn_count != 0 && ssn_count >= ctx->engine->min_ssn_count) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scan_structured: %u social security numbers detected\n", ssn_count);
Strong indicator precedence over PUA / Heuristic detections Signatures that start with "PUA.", "Heuristics.", or "BC.Heuristics." are perceived to be less serious, or more likely to have false positives, than other signatures that we would think of us as "strong indicators". At present, only a subset of "Heuristics." signatures, such as those added by the phishing module, are added as "potentially unwanted". Unless you're using heuristic-precedence mode, these "potentially unwanted" indicators are recorded but not reported unless no other signature alerts. This behavior should apply to all signatures that start with "PUA." and "Heuristics.". We already do a string match comparison on the signature name to apply that behavior to bytecode matches that start with "BC.Heuristics.". I moved that string comparison logic used for "BC.Heuristics." into the main `cl_append_virus()` function and extended it to cover the other two cases. I also replaced all hardcoded calls to append "Heuristics." signatures to append using the `cli_append_potentially_unwanted()` function, so we can skip the string compare in these cases. That function will of course append them as strong indicators if heuristic-precedence mode is enabled.
2022-08-19 16:21:42 -07:00
if (CL_VIRUS == cli_append_potentially_unwanted(ctx, "Heuristics.Structured.SSN")) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return CL_VIRUS;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
}
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
return CL_SUCCESS;
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
}
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
static cl_error_t cli_scanembpe(cli_ctx *ctx, off_t offset)
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
int fd;
size_t bytes;
size_t size = 0;
size_t todo;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
const char *buff;
char *tmpname;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fmap_t *map = ctx->fmap;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
unsigned int corrupted_input;
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
tmpname = cli_gentemp_with_prefix(ctx->this_layer_tmpdir, "embedded-pe");
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!tmpname)
return CL_EMEM;
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
Change permission for new tmp files from RWX to RW
2020-03-30 20:42:44 -04:00
if ((fd = open(tmpname, O_RDWR | O_CREAT | O_TRUNC | O_BINARY, S_IRUSR | S_IWUSR)) < 0) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("cli_scanembpe: Can't create file %s\n", tmpname);
free(tmpname);
return CL_ECREAT;
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
}
use fmap in embpe
2010-07-29 03:55:24 +02:00
todo = map->len - offset;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while (1) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
bytes = MIN(todo, map->pgsz);
if (!bytes)
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!(buff = fmap_need_off_once(map, offset + size, bytes))) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
}
free(tmpname);
return CL_EREAD;
}
size += bytes;
todo -= bytes;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (cli_checklimits("cli_scanembpe", ctx, size, 0, 0) != CL_SUCCESS)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cli_writen(fd, buff, bytes) != bytes) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_dbgmsg("cli_scanembpe: Can't write to temporary file\n");
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
}
free(tmpname);
return CL_EWRITE;
}
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Setting ctx->corrupted_input will prevent the PE parser from reporting "broken executable" for unpacked/reconstructed files that may not be 100% to spec.
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
corrupted_input = ctx->corrupted_input;
bb#2307
2010-10-18 13:23:51 +02:00
ctx->corrupted_input = 1;
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_desc(fd, tmpname, ctx, NULL, LAYER_ATTRIBUTES_NONE);
bb#2307
2010-10-18 13:23:51 +02:00
ctx->corrupted_input = corrupted_input;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (ret != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
}
free(tmpname);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
return ret;
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
}
close(fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tmpname)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
free(tmpname);
return CL_EUNLINK;
}
hardfail on unlink failure git-svn: trunk@3775
2008-04-08 17:45:05 +00:00
}
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
free(tmpname);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
return CL_SUCCESS;
extract and scan PE files embedded into other executables or fake zip files generated by some worms git-svn: trunk@2934
2007-03-12 21:31:40 +00:00
}
Correction to enabling dev-performance bug introduced with the scan options change, and enablig dev-performance on Mac.
2018-10-19 20:43:19 -07:00
#if defined(_WIN32) || defined(C_LINUX) || defined(C_DARWIN)
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
#define PERF_MEASURE
#endif
#ifdef PERF_MEASURE
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static struct
{
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
enum perfev id;
const char *name;
enum ev_type type;
} perf_events[] = {
{PERFT_SCAN, "full scan", ev_time},
{PERFT_PRECB, "prescan cb", ev_time},
{PERFT_POSTCB, "postscan cb", ev_time},
{PERFT_CACHE, "cache", ev_time},
{PERFT_FT, "filetype", ev_time},
{PERFT_CONTAINER, "container", ev_time},
{PERFT_SCRIPT, "script", ev_time},
{PERFT_PE, "pe", ev_time},
{PERFT_RAW, "raw", ev_time},
{PERFT_RAWTYPENO, "raw container", ev_time},
{PERFT_MAP, "map", ev_time},
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
{PERFT_BYTECODE, "bytecode", ev_time},
{PERFT_KTIME, "kernel", ev_int},
{PERFT_UTIME, "user", ev_int}};
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
static void get_thread_times(uint64_t *kt, uint64_t *ut)
{
#ifdef _WIN32
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
FILETIME c, e, k, u;
ULARGE_INTEGER kl, ul;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!GetThreadTimes(GetCurrentThread(), &c, &e, &k, &u)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
*kt = *ut = 0;
return;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
kl.LowPart = k.dwLowDateTime;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
kl.HighPart = k.dwHighDateTime;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ul.LowPart = u.dwLowDateTime;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
ul.HighPart = u.dwHighDateTime;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
*kt = kl.QuadPart / 10;
*ut = ul.QuadPart / 10;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
#else
struct tms tbuf;
Assortment of warning fixes.
2019-05-03 18:25:17 -04:00
if (times(&tbuf) != ((clock_t)-1)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
clock_t tck = sysconf(_SC_CLK_TCK);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
*kt = ((uint64_t)1000000) * tbuf.tms_stime / tck;
*ut = ((uint64_t)1000000) * tbuf.tms_utime / tck;
} else {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
*kt = *ut = 0;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
}
#endif
}
static inline void perf_init(cli_ctx *ctx)
{
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
uint64_t kt, ut;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
unsigned i;
Restructured scan options flags from a single bitflag field to a structure containing multiple bitflag fields. This also required adding a new function to the bytecode API to get scan options a la carte, and modifying the existing function to hand back scan options in the old/deprecated uint32_t bitflag format. Re-generated bytecode iface header files. Updated libclamav documentation detailing new scan options structure. Renamed references to 'algorithmic' detection to 'heuristic' detection. Renaming references to 'properties' to 'collect metadata'. Renamed references to 'scan all' to 'scan all match'. Renamed a couple of 'Hueristic.*' signature names as 'Heuristics.*' signatures (plural) to match majority of other heuristics.
2018-07-20 22:28:48 -04:00
if (!SCAN_DEV_COLLECT_PERF_INFO)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
ctx->perf = cli_events_new(PERFT_LAST);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
for (i = 0; i < sizeof(perf_events) / sizeof(perf_events[0]); i++) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (cli_event_define(ctx->perf, perf_events[i].id, perf_events[i].name,
perf_events[i].type, multiple_sum) == -1)
continue;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
}
cli_event_time_start(ctx->perf, PERFT_SCAN);
get_thread_times(&kt, &ut);
cli_event_int(ctx->perf, PERFT_KTIME, -kt);
cli_event_int(ctx->perf, PERFT_UTIME, -ut);
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static inline void perf_done(cli_ctx *ctx)
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
{
char timestr[512];
char *p;
unsigned i;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
uint64_t kt, ut;
fix another win32 build error
2011-02-14 19:25:22 +02:00
char *pend;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
cli_events_t *perf = ctx->perf;
if (!perf)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
p = timestr;
pend = timestr + sizeof(timestr) - 1;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
*pend = 0;
cli_event_time_stop(perf, PERFT_SCAN);
get_thread_times(&kt, &ut);
cli_event_int(perf, PERFT_KTIME, kt);
cli_event_int(perf, PERFT_UTIME, ut);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
for (i = 0; i < sizeof(perf_events) / sizeof(perf_events[0]); i++) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
union ev_val val;
unsigned count;
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_event_get(perf, perf_events[i].id, &val, &count);
if (p < pend)
p += snprintf(p, pend - p, "%s: %d.%03ums, ", perf_events[i].name,
(signed)(val.v_int / 1000),
(unsigned)(val.v_int % 1000));
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
}
*p = 0;
cli_infomsg(ctx, "performance: %s\n", timestr);
cli_events_free(perf);
ctx->perf = NULL;
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static inline void perf_start(cli_ctx *ctx, int id)
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
{
cli_event_time_start(ctx->perf, id);
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static inline void perf_stop(cli_ctx *ctx, int id)
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
{
cli_event_time_stop(ctx->perf, id);
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static inline void perf_nested_start(cli_ctx *ctx, int id, int nestedid)
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
{
cli_event_time_nested_start(ctx->perf, id, nestedid);
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static inline void perf_nested_stop(cli_ctx *ctx, int id, int nestedid)
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
{
cli_event_time_nested_stop(ctx->perf, id, nestedid);
}
#else
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
static inline void perf_init(cli_ctx *ctx)
{
UNUSEDPARAM(ctx);
}
static inline void perf_start(cli_ctx *ctx, int id)
{
UNUSEDPARAM(ctx);
UNUSEDPARAM(id);
}
static inline void perf_stop(cli_ctx *ctx, int id)
{
UNUSEDPARAM(ctx);
UNUSEDPARAM(id);
}
static inline void perf_nested_start(cli_ctx *ctx, int id, int nestedid)
{
UNUSEDPARAM(ctx);
UNUSEDPARAM(id);
UNUSEDPARAM(nestedid);
}
static inline void perf_nested_stop(cli_ctx *ctx, int id, int nestedid)
{
UNUSEDPARAM(ctx);
UNUSEDPARAM(id);
UNUSEDPARAM(nestedid);
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
static inline void perf_done(cli_ctx *ctx)
{
UNUSEDPARAM(ctx);
}
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
#endif
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
/**
* @brief Perform raw scan of current fmap.
*
libclamav, clamscan: load/unload callbacks & progress meters Add progress callbacks to libclamav for: - database load - engine compile - engine free Add a progress bar to clamscan for load & compile. These are disabled if you run with --debug or stdout is not a TTY or you are using one of --quiet, --infected, or --no-summary. Added code so you can test the engine-free callback by building with ENABLE_ENGINE_FREE_PROGRESSBAR defined. The compile & free progress callbacks pre-calculate the number of tasks to complete to estimate the progress. Some tasks may take longer than others so the progress speed my appear to vary a little. The callbacks return type is a cl_error_t but doesn't currently do anything. It is reserved for future use. Minor formatting change in matcher-ac.c to counteract weird clang-format behavior, and to make it easier to read. Added progress callbacks and clamscan progress bars to the news.
2021-07-16 11:47:23 -07:00
* @param ctx Current scan context.
* @param type File type
* @param typercg Enable type recognition (file typing scan results).
* If 0, will be a regular ac-mode scan.
* @param[out] dettype If typercg enabled and scan detects HTML or MAIL types,
* will output HTML or MAIL types after performing HTML/MAIL scans
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
* @return cl_error_t
*/
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
static cl_error_t scanraw(cli_ctx *ctx, cli_file_t type, uint8_t typercg, cli_file_t *dettype)
add support for signature based self-protection mode git-svn: trunk@1831
2006-01-25 12:11:31 +00:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS, nret = CL_SUCCESS;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
struct cli_matched_type *ftoffset = NULL, *fpt;
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
unsigned int acmode = AC_SCAN_VIR;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_file_t found_type;
Disable embedded type recognition for disk images Using file type recognition scan mode for disk images and other raw archive formats is problematic. One simple reason is that the contained files will be detected and parsed and scanned twice, first when deteced by the type recog scan, and later when the archive is extracted and the files are properly scanned. Another reason is an increased likelihood for incorrect type recognition, as seen with supposed MHTML files (they weren't) found in GPT disk images. Though a previous patch disabled embedded type recognition for GPT files, this one extens this to the following: - CL_TYPE_CPIO_OLD - CL_TYPE_ZIP - CL_TYPE_OLD_TAR - CL_TYPE_POSIX_TAR ZIP is included because file entries in a ZIP are incorrectly detected as ZIPSFX's and though we also ensure not to scan ZIPSFX's found in ZIP's, it's more efficient not to do the type recognition in the first place and it prevents us from adding those bogus ZIPSFX entries into the scan properties JSON. This patch also fixes what appears to be a copy-paste typo, where CL_TYPE_ISHIELD_MSI types were accidentally having their container value set to CL_TYPE_AUTOIT.
2020-08-12 00:18:53 -07:00
if ((typercg) &&
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
// We should also omit bzips, but DMG's may be detected in bzips. (type != CL_TYPE_BZ) && /* Omit BZ files because they can contain portions of original files like zip file entries that cause invalid extractions and lots of warnings. Decompress first, then scan! */
Fix double-extraction of OOXML-based office documents File-type scanning for OOXML-based office documents results in double-extraction, because we also extract OOXML document components in the appopriate parser modules. CLAM-1412
2025-06-16 20:11:53 -04:00
(type != CL_TYPE_GZ) && /* Omit GZ files because they can contain portions of original files like zip file entries that cause invalid extractions and lots of warnings. Decompress first, then scan! */
(type != CL_TYPE_CPIO_OLD) && /* Omit CPIO_OLD files because it's an image format that we can extract and scan manually. */
(type != CL_TYPE_ZIP) && /* Omit ZIP files because it'll detect each zip file entry as SFXZIP, which is a waste. We'll extract it and then scan. */
(type != CL_TYPE_ZIPSFX) && /* Omit SFX archive types from being checked for embedded content. They should only be parsed for contained files. Those contained files could be EXE's with more SFX, but that's the nature of containers. */
(type != CL_TYPE_ARJSFX) && /* " */
(type != CL_TYPE_RARSFX) && /* " */
(type != CL_TYPE_EGGSFX) && /* " */
(type != CL_TYPE_CABSFX) && /* " */
(type != CL_TYPE_7ZSFX) && /* " */
(type != CL_TYPE_OOXML_WORD) && /* Omit OOXML because they are ZIP-based and file-type scanning will double-extract their contents. */
(type != CL_TYPE_OOXML_PPT) && /* " */
(type != CL_TYPE_OOXML_XL) && /* " */
(type != CL_TYPE_OOXML_HWP) && /* " */
(type != CL_TYPE_OLD_TAR) && /* Omit OLD TAR files because it's a raw archive format that we can extract and scan manually. */
(type != CL_TYPE_POSIX_TAR)) { /* Omit POSIX TAR files because it's a raw archive format that we can extract and scan manually. */
Disable embedded type recognition for disk images Using file type recognition scan mode for disk images and other raw archive formats is problematic. One simple reason is that the contained files will be detected and parsed and scanned twice, first when deteced by the type recog scan, and later when the archive is extracted and the files are properly scanned. Another reason is an increased likelihood for incorrect type recognition, as seen with supposed MHTML files (they weren't) found in GPT disk images. Though a previous patch disabled embedded type recognition for GPT files, this one extens this to the following: - CL_TYPE_CPIO_OLD - CL_TYPE_ZIP - CL_TYPE_OLD_TAR - CL_TYPE_POSIX_TAR ZIP is included because file entries in a ZIP are incorrectly detected as ZIPSFX's and though we also ensure not to scan ZIPSFX's found in ZIP's, it's more efficient not to do the type recognition in the first place and it prevents us from adding those bogus ZIPSFX entries into the scan properties JSON. This patch also fixes what appears to be a copy-paste typo, where CL_TYPE_ISHIELD_MSI types were accidentally having their container value set to CL_TYPE_AUTOIT.
2020-08-12 00:18:53 -07:00
/*
Fix typos (no functional changes)
2023-11-26 15:01:19 -08:00
* Enable file type recognition scan mode if requested, except for some problematic types (above).
Disable embedded type recognition for disk images Using file type recognition scan mode for disk images and other raw archive formats is problematic. One simple reason is that the contained files will be detected and parsed and scanned twice, first when deteced by the type recog scan, and later when the archive is extracted and the files are properly scanned. Another reason is an increased likelihood for incorrect type recognition, as seen with supposed MHTML files (they weren't) found in GPT disk images. Though a previous patch disabled embedded type recognition for GPT files, this one extens this to the following: - CL_TYPE_CPIO_OLD - CL_TYPE_ZIP - CL_TYPE_OLD_TAR - CL_TYPE_POSIX_TAR ZIP is included because file entries in a ZIP are incorrectly detected as ZIPSFX's and though we also ensure not to scan ZIPSFX's found in ZIP's, it's more efficient not to do the type recognition in the first place and it prevents us from adding those bogus ZIPSFX entries into the scan properties JSON. This patch also fixes what appears to be a copy-paste typo, where CL_TYPE_ISHIELD_MSI types were accidentally having their container value set to CL_TYPE_AUTOIT.
2020-08-12 00:18:53 -07:00
*/
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
acmode |= AC_SCAN_FT;
Disable embedded type recognition for disk images Using file type recognition scan mode for disk images and other raw archive formats is problematic. One simple reason is that the contained files will be detected and parsed and scanned twice, first when deteced by the type recog scan, and later when the archive is extracted and the files are properly scanned. Another reason is an increased likelihood for incorrect type recognition, as seen with supposed MHTML files (they weren't) found in GPT disk images. Though a previous patch disabled embedded type recognition for GPT files, this one extens this to the following: - CL_TYPE_CPIO_OLD - CL_TYPE_ZIP - CL_TYPE_OLD_TAR - CL_TYPE_POSIX_TAR ZIP is included because file entries in a ZIP are incorrectly detected as ZIPSFX's and though we also ensure not to scan ZIPSFX's found in ZIP's, it's more efficient not to do the type recognition in the first place and it prevents us from adding those bogus ZIPSFX entries into the scan properties JSON. This patch also fixes what appears to be a copy-paste typo, where CL_TYPE_ISHIELD_MSI types were accidentally having their container value set to CL_TYPE_AUTOIT.
2020-08-12 00:18:53 -07:00
}
add support for signature based self-protection mode git-svn: trunk@1831
2006-01-25 12:11:31 +00:00
Disable embedded type recognition for disk images Using file type recognition scan mode for disk images and other raw archive formats is problematic. One simple reason is that the contained files will be detected and parsed and scanned twice, first when deteced by the type recog scan, and later when the archive is extracted and the files are properly scanned. Another reason is an increased likelihood for incorrect type recognition, as seen with supposed MHTML files (they weren't) found in GPT disk images. Though a previous patch disabled embedded type recognition for GPT files, this one extens this to the following: - CL_TYPE_CPIO_OLD - CL_TYPE_ZIP - CL_TYPE_OLD_TAR - CL_TYPE_POSIX_TAR ZIP is included because file entries in a ZIP are incorrectly detected as ZIPSFX's and though we also ensure not to scan ZIPSFX's found in ZIP's, it's more efficient not to do the type recognition in the first place and it prevents us from adding those bogus ZIPSFX entries into the scan properties JSON. This patch also fixes what appears to be a copy-paste typo, where CL_TYPE_ISHIELD_MSI types were accidentally having their container value set to CL_TYPE_AUTOIT.
2020-08-12 00:18:53 -07:00
perf_start(ctx, PERFT_RAW);
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_scan_fmap(ctx, type == CL_TYPE_TEXT_ASCII ? CL_TYPE_ANY : type, false, &ftoffset, acmode, NULL);
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
perf_stop(ctx, PERFT_RAW);
add support for signature based self-protection mode git-svn: trunk@1831
2006-01-25 12:11:31 +00:00
Fix typos (no functional changes)
2024-01-19 09:08:36 -08:00
// In allmatch-mode, ret will never be CL_VIRUS, so ret may be used exclusively for file type detection and for terminal errors.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// When not in allmatch-mode, it's more important to return right away if ret is CL_VIRUS, so we don't care if file type matches were found.
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ret >= CL_TYPENO) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Matched 1+ file type signatures. Handle them.
found_type = (cli_file_t)ret;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
perf_nested_start(ctx, PERFT_RAWTYPENO, PERFT_SCAN);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
fpt = ftoffset;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while (fpt) {
Add EmbeddedObjects to JSON The metadata projecties JSON structure isn't recording file types found embedded within a file such as self-extracting (SFX) types and office document types (DOCX, PPTX, etc). This presents a problem... At present there's no way to know if the current file has ended and a few file is found tacked on to the end of the first file. If there were, we could simply check if the type found by the raw-scan exists within the first file, or after. If within the first, and the type is an archive then it's reasonable to conclude we're either observing zip headers (for SFXZIP detections) or other files that are not compressed. If the type ISN'T found within the first file, then we definitely have whole new file to parse and we should do so with cli_magic_scan() rather than only using these embedded type scanners. At present we can't ignore SFXZIP detections even if the original file type is a ZIP because we may have found two ZIPs appended together to evade detection (a legitimate trick). As a consequence, we will effectively parse every zip entry twice. The same issue applies to types found within non-compressed archives. This commit adds an EmbeddedObjects list to the metadata JSON object so that the existance of these types is noted. Additionally, this commit removes the two-part int64 cli_jsonint64() implementation as json_object_new_int64() should be available everywhere and the macro to detect such support was never set.
2020-03-21 11:36:53 -04:00
if (fpt->offset > 0) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
bool type_has_been_handled = true;
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
bool ancestor_was_embedded = false;
size_t i;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
/*
* First, use "embedded type recognition" to identify a file's actual type.
* (a.k.a. not embedded files, but file type detection corrections)
*
* Do this at all fmap layers. Though we should only reassign the types
* if the current type makes sense for the reassignment.
*/
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
switch (fpt->type) {
case CL_TYPE_MHTML:
if (SCAN_PARSE_MAIL && (DCONF_MAIL & MAIL_CONF_MBOX)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((ctx->recursion_stack[ctx->recursion_level].type >= CL_TYPE_TEXT_ASCII) &&
(ctx->recursion_stack[ctx->recursion_level].type <= CL_TYPE_BINARY_DATA)) {
// HTML files may contain special characters and could be
// misidentified as BINARY_DATA by cli_compare_ftm_file()
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("MHTML signature found at %u\n", (unsigned int)fpt->offset);
nret = ret = cli_scanmail(ctx);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
MHTML: added filetype and switch case
2016-05-02 17:32:03 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_XDP:
if (SCAN_PARSE_PDF && (DCONF_DOC & DOC_CONF_PDF)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((ctx->recursion_stack[ctx->recursion_level].type >= CL_TYPE_TEXT_ASCII) &&
(ctx->recursion_stack[ctx->recursion_level].type <= CL_TYPE_BINARY_DATA)) {
// XML files may contain special characters and could be
// misidentified as BINARY_DATA by cli_compare_ftm_file()
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("XDP signature found at %u\n", (unsigned int)fpt->offset);
nret = ret = cli_scanxdp(ctx);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_XML_WORD:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_MSXML)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((ctx->recursion_stack[ctx->recursion_level].type >= CL_TYPE_TEXT_ASCII) &&
(ctx->recursion_stack[ctx->recursion_level].type <= CL_TYPE_BINARY_DATA)) {
// XML files may contain special characters and could be
// misidentified as BINARY_DATA by cli_compare_ftm_file()
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("XML-WORD signature found at %u\n", (unsigned int)fpt->offset);
nret = ret = cli_scanmsxml(ctx);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
case CL_TYPE_XML_XL:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_MSXML)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((ctx->recursion_stack[ctx->recursion_level].type >= CL_TYPE_TEXT_ASCII) &&
(ctx->recursion_stack[ctx->recursion_level].type <= CL_TYPE_BINARY_DATA)) {
// XML files may contain special characters and could be
// misidentified as BINARY_DATA by cli_compare_ftm_file()
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("XML-XL signature found at %u\n", (unsigned int)fpt->offset);
nret = ret = cli_scanmsxml(ctx);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
case CL_TYPE_XML_HWP:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_HWP)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((ctx->recursion_stack[ctx->recursion_level].type >= CL_TYPE_TEXT_ASCII) &&
(ctx->recursion_stack[ctx->recursion_level].type <= CL_TYPE_BINARY_DATA)) {
// XML files may contain special characters and could be
// misidentified as BINARY_DATA by cli_compare_ftm_file()
Fix file access issue if in low privelege process Removing problematic call to convert file descriptors to filepaths. Added filename and tempfile names to scandesc calls in clamd. Added a general scan option to treat the scan engine as unprivileged, meaning that the scan engine will not have read access to the file. Added check to drop a temp file for RAR's where the we don't have read access to the filepath provided (i.e. unprivileged is set, or access() check fails).
2020-01-23 17:42:33 -08:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("XML-HWP signature found at %u\n", (unsigned int)fpt->offset);
nret = ret = cli_scanhwpml(ctx);
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
bb #9576: add fmap_dump_to_file function
2013-11-22 19:41:46 -05:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
break;
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
case CL_TYPE_DMG:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_DMG)) {
// TODO: determine all types that DMG may start with
// if ((ctx->recursion_stack[ctx->recursion_level].type == CL_TYPE_BZIP2) || ...))
{
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("DMG signature found at %u\n", (unsigned int)fpt->offset);
nret = cli_scandmg(ctx);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
}
break;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
case CL_TYPE_ISO9660:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ISO9660)) {
// TODO: determine all types that ISO9660 may start with
// if ((ctx->recursion_stack[ctx->recursion_level].type == CL_TYPE_ANY) || ...))
{
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("ISO signature found at %u\n", (unsigned int)fpt->offset);
nret = cli_scaniso(ctx, fpt->offset);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
Add support for UDF files Add support for specifically for Beginning Extended Area Descriptor (BEA01) type of UDF files.
2023-06-02 10:45:38 -07:00
case CL_TYPE_UDF:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_UDF)) {
{
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, fpt->type, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("UDF signature found at %u\n", (unsigned int)fpt->offset);
nret = cli_scanudf(ctx, fpt->offset);
}
Add support for UDF files Add support for specifically for Beginning Extended Area Descriptor (BEA01) type of UDF files.
2023-06-02 10:45:38 -07:00
}
}
break;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
case CL_TYPE_MBR:
if (SCAN_PARSE_ARCHIVE) {
// TODO: determine all types that GPT or MBR may start with
// if ((ctx->recursion_stack[ctx->recursion_level].type == CL_TYPE_???) || ...))
{
// First check if actually a GPT, not MBR.
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
cl_error_t iret = cli_mbr_check2(ctx, 0);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((iret == CL_TYPE_GPT) && (DCONF_ARCH & ARCH_CONF_GPT)) {
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, CL_TYPE_GPT, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("Recognized GUID Partition Table file\n");
cli_dbgmsg("GPT signature found at %u\n", (unsigned int)fpt->offset);
nret = cli_scangpt(ctx, 0);
}
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
} else if ((iret == CL_SUCCESS) && (DCONF_ARCH & ARCH_CONF_MBR)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
// Reassign type of current layer based on what we discovered
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, CL_TYPE_MBR, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
type_has_been_handled = false;
} else {
cli_dbgmsg("MBR signature found at %u\n", (unsigned int)fpt->offset);
nret = cli_scanmbr(ctx, 0);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
default:
type_has_been_handled = false;
}
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if ((CL_EMEM == nret) || ctx->abort_scan) {
break;
}
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
/*
* Only scan embedded files if we are not already in an embedded context.
* That is, if this or a previous layer was identified with embedded file type recognition, then we do
* not scan for embedded files again.
*
* This restriction will prevent detecting the same embedded content more than once when recursing with
* embedded file type recognition deeper within the same buffer.
*
* This is necessary because we have no way of knowing the length of a file and cannot prevent a search
* for embedded files from finding the same embedded content multiple times (like a LOT of times).
*
* E.g. if the file is like this:
*
* [ data ] [ embedded file ] [ data ] [ embedded file ]
*
* The first time we do it we'll find "two" embedded files, like this:
*
* Emb. File #1: [ embedded file ] [ data ] [ embedded file ]
* Emb. File #2: [ embedded file ]
*
* We must not scan Emb. File #1 again for embedded files, because it would double-extract Emb. File #2.
*
* There is a flaw in this logic, though. Suppose that we actually have:
*
* [ data ] [ compressed file w. recognizable magic bytes ]
*
* A first pass of the above will again identify "two" embedded files:
*
* Emb. File #1: [ compressed archive w. recognizable magic bytes ]
* Emb. File #2: [ magic bytes ] <- Compressed data/Not real file
*
* In this case, the magic bytes of a contained, compressed file is somehow still identifiable despite
* compression. The result is the Emb. File #2 will fail to be parsed and when we decompress Emb. File
* #1, then we maybe get something like this:
*
* Decompressed: [ data ] [ embedded file ]
*
* So if this happened... then we WOULD want to scan the decompressed file for embedded files.
* The problem is, we have way of knowing how long embedded files are.
* We don't know if we have:
*
* A. [ data ] [ embedded file ] [ data ] [ embedded file ]
* or
* B. [ data ] [ embedded compressed archive w. recognizable magic bytes ]
* or
* C. [ data ] [ embedded uncompressed archive w. multiple file entries [ file 1 ] [ file 2 ] [ file 2 ] ]
*
* Some ideas for a more accurate solution:
*
* 1. Record the offset and size of each file extracted by the parsers.
* Then, when we do embedded file type recognition, we can check if the offset and size of the
* embedded file matches the offset and size of a file that was extracted by a parser.
* This falls apart a little bit for multiple layers of archives unless we also compare offsets within
* each layer. We could do that, but it would be a lot of work. And we'd probably want to take into
* consideration if files were decompressed or decrypted. ... I don't know a clean solution.
*
* 2. Have all parsers to run before embedded file type recognition and they each determine the length
* of the file they parsed, so we can differentiate between embedded files and appended files.
* For appended files, we would know they weren't extracted by a parser module and the parser for
* each of those would report the length of the file it parsed so we can use that to mitigate
* overlapping embedded file type recognition.
* But I highly doubt all file types can be parsed to determine the correct length of the file.
*/
for (i = ctx->recursion_level; i > 0; i--) {
if (ctx->recursion_stack[i].attributes & LAYER_ATTRIBUTES_EMBEDDED) {
// Found an ancestor that was embedded.
// Do not scan embedded files again.
ancestor_was_embedded = true;
break;
}
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
/*
* Next, check for actual embedded files.
*/
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if ((false == ancestor_was_embedded) &&
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
(false == type_has_been_handled)) {
cli_dbgmsg("%s signature found at %u\n", cli_ftname(fpt->type), (unsigned int)fpt->offset);
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
type_has_been_handled = true;
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
switch (fpt->type) {
case CL_TYPE_RARSFX:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_RAR) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_RAR,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
break;
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
case CL_TYPE_EGGSFX:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_EGG) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_EGG,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
break;
case CL_TYPE_ZIPSFX:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_ZIP) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_ZIP,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
break;
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
case CL_TYPE_CABSFX:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_MSCAB) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_MSCAB,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_ARJSFX:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_ARJ) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_ARJ,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_7ZSFX:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_7Z) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_7Z,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_NULSFT:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type == CL_TYPE_MSEXE && fpt->offset > 4) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
// Note: CL_TYPE_NULSFT is special, because the file actually starts 4 bytes before the start of the signature match
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset - 4,
ctx->fmap->len - (fpt->offset - 4),
ctx,
CL_TYPE_NULSFT,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_AUTOIT:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type == CL_TYPE_MSEXE) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_AUTOIT,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_ISHIELD_MSI:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type == CL_TYPE_MSEXE) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_ISHIELD_MSI,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_PDF:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type != CL_TYPE_PDF) {
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_PDF,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
case CL_TYPE_MSEXE:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (type == CL_TYPE_MSEXE || type == CL_TYPE_ZIP || type == CL_TYPE_MSOLE2) {
cli_dbgmsg("*** Detected embedded PE file at %u ***\n", (unsigned int)fpt->offset);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if ((uint64_t)(ctx->fmap->len - fpt->offset) > ctx->engine->maxembeddedpe) {
cli_dbgmsg("scanraw: MaxEmbeddedPE exceeded\n");
break;
}
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
nret = cli_magic_scan_nested_fmap_type(
ctx->fmap,
fpt->offset,
ctx->fmap->len - fpt->offset,
ctx,
CL_TYPE_MSEXE,
NULL,
LAYER_ATTRIBUTES_EMBEDDED);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
break;
default:
type_has_been_handled = false;
cli_dbgmsg("scanraw: Type %u not handled in fpt loop\n", fpt->type);
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
} // end check for embedded files
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
libclamav: Add URI scanning support to PDF parser Threat Research requests scanning URIs in PDF files and adding them to the json report file. This change adds URI scanning support to the PDF parser, including support for object references to URIs in PDF files. Jira: CLAM-2588 Fix out-of-order references and other minor improvements. CLAM-2588, CLAM-2757
2025-04-07 16:50:09 -07:00
} // end if (fpt->offset > 0)
better handling of embedded stuff git-svn: trunk@2965
2007-03-23 15:11:45 +00:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if ((nret == CL_EMEM) ||
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
(ctx->abort_scan)) {
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
break;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
better handling of embedded stuff git-svn: trunk@2965
2007-03-23 15:11:45 +00:00
Allow html detection/normalization for graphic file types. Remove dead code.
2013-10-23 16:21:46 -04:00
fpt = fpt->next;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
} // end while (fpt) loop
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (!((nret == CL_EMEM) || (ctx->abort_scan))) {
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* Now run the other file type parsers that may rely on file type
* recognition to determine the actual file type.
*/
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
switch (found_type) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_HTML:
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (cli_recursion_stack_get_type(ctx, -2) == CL_TYPE_AUTOIT) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/* bb#11196 - autoit script file misclassified as HTML */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ret = CL_TYPE_TEXT_ASCII;
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
} else if (SCAN_PARSE_HTML &&
(type == CL_TYPE_TEXT_ASCII ||
type == CL_TYPE_GIF) && /* Scan GIFs for embedded HTML/Javascript */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
(DCONF_DOC & DOC_CONF_HTML)) {
*dettype = CL_TYPE_HTML;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, CL_TYPE_HTML, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
} else {
nret = cli_scanhtml(ctx);
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_MAIL:
if (SCAN_PARSE_MAIL && type == CL_TYPE_TEXT_ASCII && (DCONF_MAIL & MAIL_CONF_MBOX)) {
*dettype = CL_TYPE_MAIL;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (CL_SUCCESS != (ret = cli_recursion_stack_change_type(ctx, CL_TYPE_MAIL, true))) {
cli_dbgmsg("Call to cli_recursion_stack_change_type() returned %s \n", cl_strerror(ret));
} else {
nret = cli_scanmail(ctx);
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
default:
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
perf_nested_stop(ctx, PERFT_RAWTYPENO, PERFT_SCAN);
ret = nret;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
} // end if (ret >= CL_TYPENO)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
while (ftoffset) {
fpt = ftoffset;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
ftoffset = ftoffset->next;
free(fpt);
}
add support for signature based self-protection mode git-svn: trunk@1831
2006-01-25 12:11:31 +00:00
return ret;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
void emax_reached(cli_ctx *ctx)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
{
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
int32_t stack_index;
if (NULL == ctx || NULL == ctx->recursion_stack) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return;
set dont_cache_flag on the fmap stack
2010-03-05 18:09:00 +01:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
stack_index = (int32_t)ctx->recursion_level;
while (stack_index >= 0) {
fmap_t *map = ctx->recursion_stack[stack_index].fmap;
if (NULL != map) {
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
map->dont_cache_flag = true;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
stack_index -= 1;
}
set dont_cache_flag on the fmap stack
2010-03-05 18:09:00 +01:00
cli_dbgmsg("emax_reached: marked parents as non cacheable\n");
}
add spam to cache and magicscan
2010-03-05 17:11:45 +01:00
#define LINESTR(x) #x
#define LINESTR2(x) LINESTR(x)
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
#define __AT__ " at line " LINESTR2(__LINE__)
fix wrong interaction between prescan_cb caching and postscan_cb This bug doesn't affect clamd, and clamscan, since they don't use the callbacks. postscan_cb could return CL_VIRUS, but we've already cache_add-ed as CLEAN. Thus on next scan we'll immediately return CL_CLEAN, instead of invoking any of the callbacks. Fix this by caching the result as CLEAN only after postscan_cb was invoked (if set). The CL_BREAK result from postcan_cb is not cached as CLEAN, because it might refer to a previous inner file we scanned, not necesarely the current one. Also make sure postscan is only invoked _after_ prescan was invoked, for early returns don't invoke postscan at all.
2011-07-04 17:00:55 +03:00
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
/**
* @brief Provide the following to the calling application for each embedded file:
* - name of parent file
* - size of parent file
* - name of current file
* - size of current file
* - pointer to the current file data
*
* @param cb
* @param ctx
* @param filetype
* @return cl_error_t
*/
static cl_error_t dispatch_file_inspection_callback(clcb_file_inspection cb, cli_ctx *ctx, const char *filetype)
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cl_error_t append_ret;
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
int fd = -1;
uint32_t fmap_index = ctx->recursion_level; /* index of current file */
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
cl_fmap_t *fmap = NULL;
const char *file_name = NULL;
size_t file_size = 0;
const char *file_buffer = NULL;
const char **ancestors = NULL;
size_t parent_file_size = 0;
if (NULL == cb) {
// Callback is not set.
goto done;
}
fmap = ctx->recursion_stack[fmap_index].fmap;
fd = fmap_fd(fmap);
Refine max-allocation and safer-allocation function and macro names We add the _OR_GOTO_DONE suffix to the macros that go to done if the allocation fails. This makes it obvious what is different about the macro versus the equivalent function, and that error handling is built-in. Renamed the cli_strdup to safer_strdup to make it obvious that it exists because it is safer than regular strdup. Regular strdup doesn't have the NULL check before trying to dup, and so may result in a NULL-deref crash. Also remove unused STRDUP (_OR_GOTO_DONE) macro, since the one with the NULL-check is preferred.
2024-01-09 17:44:33 -05:00
CLI_MAX_CALLOC_OR_GOTO_DONE(ancestors, ctx->recursion_level + 1, sizeof(char *), status = CL_EMEM);
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
file_name = fmap->name;
file_buffer = fmap_need_off_once_len(fmap, 0, fmap->len, &file_size);
while (fmap_index > 0) {
cl_fmap_t *previous_fmap;
fmap_index -= 1;
previous_fmap = ctx->recursion_stack[fmap_index].fmap;
if (ctx->recursion_level > 0 && (fmap_index == ctx->recursion_level - 1)) {
parent_file_size = previous_fmap->len;
}
ancestors[fmap_index] = previous_fmap->name;
}
perf_start(ctx, PERFT_INSPECT);
status = cb(fd, filetype, ancestors, parent_file_size, file_name, file_size, file_buffer,
ctx->recursion_level, ctx->recursion_stack[ctx->recursion_level].attributes, ctx->cb_ctx);
perf_stop(ctx, PERFT_INSPECT);
switch (status) {
case CL_BREAK:
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cli_dbgmsg("dispatch_file_inspection_callback: file trusted by callback\n");
// Remove any evidence for this layer and set the verdict to trusted.
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
(void)cli_trust_this_layer(ctx, "legacy file-inspection application callback");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
break;
case CL_VIRUS:
cli_dbgmsg("dispatch_file_inspection_callback: file blocked by callback\n");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
append_ret = cli_append_virus(ctx, "Detected.By.Callback.Inspection");
if (append_ret == CL_VIRUS) {
status = CL_VIRUS;
}
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
break;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
case CL_SUCCESS:
// No action requested by callback. Keep scanning.
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
break;
default:
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_SUCCESS;
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
cli_warnmsg("dispatch_file_inspection_callback: ignoring bad return code from callback\n");
}
done:
Refine max-allocation and safer-allocation function and macro names We add the _OR_GOTO_DONE suffix to the macros that go to done if the allocation fails. This makes it obvious what is different about the macro versus the equivalent function, and that error handling is built-in. Renamed the cli_strdup to safer_strdup to make it obvious that it exists because it is safer than regular strdup. Regular strdup doesn't have the NULL check before trying to dup, and so may result in a NULL-deref crash. Also remove unused STRDUP (_OR_GOTO_DONE) macro, since the one with the NULL-check is preferred.
2024-01-09 17:44:33 -05:00
CLI_FREE_AND_SET_NULL(ancestors);
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
return status;
}
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
static cl_error_t dispatch_prescan_callback(clcb_pre_scan cb, cli_ctx *ctx, const char *filetype, bool pre_cache)
Convert two macros to functions
2014-04-30 15:01:05 -04:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cl_error_t append_ret;
Convert two macros to functions
2014-04-30 15:01:05 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (cb) {
Convert two macros to functions
2014-04-30 15:01:05 -04:00
perf_start(ctx, PERFT_PRECB);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = cb(fmap_fd(ctx->fmap), filetype, ctx->cb_ctx);
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
perf_stop(ctx, PERFT_PRECB);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
switch (status) {
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
case CL_BREAK: {
const char *source = pre_cache ? "legacy pre-cache application callback"
: "legacy pre-scan application callback";
Blacklist & Whitelist verbiage Improvements to use modern block list and allow list verbiage. blacklist -> block list whitelist -> allow listed blacklisted -> blocked whitelisted -> allowed In the case of certificate verification, use "trust" or "verify" when something is allowed. Also changed domainlist -> domain list (or DomainList) to match.
2021-05-27 13:15:52 -07:00
cli_dbgmsg("dispatch_prescan_callback: file allowed by callback\n");
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
// Remove any evidence for this layer and set the verdict to trusted.
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
(void)cli_trust_this_layer(ctx, source);
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
status = CL_VERIFIED;
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
} break;
case CL_VIRUS: {
const char *alert_name = pre_cache ? "Detected.By.Callback.PreCache"
: "Detected.By.Callback.PreScan";
Blacklist & Whitelist verbiage Improvements to use modern block list and allow list verbiage. blacklist -> block list whitelist -> allow listed blacklisted -> blocked whitelisted -> allowed In the case of certificate verification, use "trust" or "verify" when something is allowed. Also changed domainlist -> domain list (or DomainList) to match.
2021-05-27 13:15:52 -07:00
cli_dbgmsg("dispatch_prescan_callback: file blocked by callback\n");
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
append_ret = cli_append_virus(ctx, alert_name);
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (append_ret == CL_VIRUS) {
status = CL_VIRUS;
}
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
} break;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
case CL_SUCCESS:
// No action requested by callback. Keep scanning.
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
break;
default:
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_SUCCESS;
Disable embedded type recognition for disk images Using file type recognition scan mode for disk images and other raw archive formats is problematic. One simple reason is that the contained files will be detected and parsed and scanned twice, first when deteced by the type recog scan, and later when the archive is extracted and the files are properly scanned. Another reason is an increased likelihood for incorrect type recognition, as seen with supposed MHTML files (they weren't) found in GPT disk images. Though a previous patch disabled embedded type recognition for GPT files, this one extens this to the following: - CL_TYPE_CPIO_OLD - CL_TYPE_ZIP - CL_TYPE_OLD_TAR - CL_TYPE_POSIX_TAR ZIP is included because file entries in a ZIP are incorrectly detected as ZIPSFX's and though we also ensure not to scan ZIPSFX's found in ZIP's, it's more efficient not to do the type recognition in the first place and it prevents us from adding those bogus ZIPSFX entries into the scan properties JSON. This patch also fixes what appears to be a copy-paste typo, where CL_TYPE_ISHIELD_MSI types were accidentally having their container value set to CL_TYPE_AUTOIT.
2020-08-12 00:18:53 -07:00
cli_warnmsg("dispatch_prescan_callback: ignoring bad return code from callback\n");
Convert two macros to functions
2014-04-30 15:01:05 -04:00
}
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
return status;
Convert two macros to functions
2014-04-30 15:01:05 -04:00
}
Extend prescan_cb with file type WARNING: THIS BREAKS THE ABI! - See clamav.h
2011-03-04 18:27:32 +01:00
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
static cl_error_t calculate_fuzzy_image_hash(cli_ctx *ctx, cli_file_t type)
{
cl_error_t status = CL_EPARSE;
const uint8_t *offset = NULL;
image_fuzzy_hash_t hash = {0};
json_object *header = NULL;
Various Rust-related code cleanup * Broke out the variants of error/result handling in `frs_error.rs`. Made syntax slightly cleaner for `frs_call!`, explicitly moving the output variables *out* of the function call so as not to make the parameter order confusing. * Wrapped the FuzzyHash map into a container rather than exposing the HashMap directly. Simplifies casting, and allows it to feel more like a class with methods. * Fixed various clippy complaints regarding unsafe, etc. * Rename `frs_error.rs` to `ffi_utils.rs` and migrated ffi-specific features like the `validate_str_param!()` macro to this new module.
2022-02-28 15:41:28 -08:00
FFIError *fuzzy_hash_calc_error = NULL;
Image fuzzy hash: improve error reporting Extends the new frs_error module to provide variants of the frs_result!() macro that accept a Result as input instead of calling a function on your behalf. This enables us to use the macro in conditions where we don't want to return on success, and want to do other things before we return. Use the new frs_error module to return errors to the C calling functions rather than logging the error in Rust-land. Notably, this enables us to store more meaningful error messages in the JSON output if we fail to calculate the image fuzzy hash.
2022-02-26 13:03:13 -08:00
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
offset = fmap_need_off(ctx->fmap, 0, ctx->fmap->real_len);
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (SCAN_COLLECT_METADATA && (NULL != ctx->this_layer_metadata_json)) {
if (NULL == (header = cli_jsonobj(ctx->this_layer_metadata_json, "ImageFuzzyHash"))) {
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
cli_errmsg("Failed to allocate ImageFuzzyHash JSON object\n");
status = CL_EMEM;
goto done;
}
}
Image fuzzy hash: improve error reporting Extends the new frs_error module to provide variants of the frs_result!() macro that accept a Result as input instead of calling a function on your behalf. This enables us to use the macro in conditions where we don't want to return on success, and want to do other things before we return. Use the new frs_error module to return errors to the C calling functions rather than logging the error in Rust-land. Notably, this enables us to store more meaningful error messages in the JSON output if we fail to calculate the image fuzzy hash.
2022-02-26 13:03:13 -08:00
if (!fuzzy_hash_calculate_image(offset, ctx->fmap->real_len, hash.hash, 8, &fuzzy_hash_calc_error)) {
cli_dbgmsg("Failed to calculate image fuzzy hash for %s: %s\n",
cli_ftname(type),
Various Rust-related code cleanup * Broke out the variants of error/result handling in `frs_error.rs`. Made syntax slightly cleaner for `frs_call!`, explicitly moving the output variables *out* of the function call so as not to make the parameter order confusing. * Wrapped the FuzzyHash map into a container rather than exposing the HashMap directly. Simplifies casting, and allows it to feel more like a class with methods. * Fixed various clippy complaints regarding unsafe, etc. * Rename `frs_error.rs` to `ffi_utils.rs` and migrated ffi-specific features like the `validate_str_param!()` macro to this new module.
2022-02-28 15:41:28 -08:00
ffierror_fmt(fuzzy_hash_calc_error));
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
if (SCAN_COLLECT_METADATA && (NULL != header)) {
Various Rust-related code cleanup * Broke out the variants of error/result handling in `frs_error.rs`. Made syntax slightly cleaner for `frs_call!`, explicitly moving the output variables *out* of the function call so as not to make the parameter order confusing. * Wrapped the FuzzyHash map into a container rather than exposing the HashMap directly. Simplifies casting, and allows it to feel more like a class with methods. * Fixed various clippy complaints regarding unsafe, etc. * Rename `frs_error.rs` to `ffi_utils.rs` and migrated ffi-specific features like the `validate_str_param!()` macro to this new module.
2022-02-28 15:41:28 -08:00
(void)cli_jsonstr(header, "Error", ffierror_fmt(fuzzy_hash_calc_error));
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
goto done;
}
if (SCAN_COLLECT_METADATA && (NULL != header)) {
char hashstr[17];
snprintf(hashstr, 17, "%02x%02x%02x%02x%02x%02x%02x%02x",
hash.hash[0], hash.hash[1], hash.hash[2], hash.hash[3],
hash.hash[4], hash.hash[5], hash.hash[6], hash.hash[7]);
(void)cli_jsonstr(header, "Hash", hashstr);
}
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
ctx->recursion_stack[ctx->recursion_level].image_fuzzy_hash = hash;
ctx->recursion_stack[ctx->recursion_level].calculated_image_fuzzy_hash = true;
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
status = CL_SUCCESS;
done:
Image fuzzy hash: improve error reporting Extends the new frs_error module to provide variants of the frs_result!() macro that accept a Result as input instead of calling a function on your behalf. This enables us to use the macro in conditions where we don't want to return on success, and want to do other things before we return. Use the new frs_error module to return errors to the C calling functions rather than logging the error in Rust-land. Notably, this enables us to store more meaningful error messages in the JSON output if we fail to calculate the image fuzzy hash.
2022-02-26 13:03:13 -08:00
if (NULL != fuzzy_hash_calc_error) {
Various Rust-related code cleanup * Broke out the variants of error/result handling in `frs_error.rs`. Made syntax slightly cleaner for `frs_call!`, explicitly moving the output variables *out* of the function call so as not to make the parameter order confusing. * Wrapped the FuzzyHash map into a container rather than exposing the HashMap directly. Simplifies casting, and allows it to feel more like a class with methods. * Fixed various clippy complaints regarding unsafe, etc. * Rename `frs_error.rs` to `ffi_utils.rs` and migrated ffi-specific features like the `validate_str_param!()` macro to this new module.
2022-02-28 15:41:28 -08:00
ffierror_free(fuzzy_hash_calc_error);
Image fuzzy hash: improve error reporting Extends the new frs_error module to provide variants of the frs_result!() macro that accept a Result as input instead of calling a function on your behalf. This enables us to use the macro in conditions where we don't want to return on success, and want to do other things before we return. Use the new frs_error module to return errors to the C calling functions rather than logging the error in Rust-land. Notably, this enables us to store more meaningful error messages in the JSON output if we fail to calculate the image fuzzy hash.
2022-02-26 13:03:13 -08:00
}
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
return status;
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/**
* @brief A unified list of reasons why a scan result inside the magic_scan function
* should goto done instead of continuing to parse/scan this layer.
*
* These are not reasons why the scan should abort entirely. For that, just check ctx->abort_scan.
*
* @param ctx The scan context.
* @param result_in The result to compare.
* @param result_out The result that magic_scan should return.
* @return true We found a reason to goto done.
* @return false The scan must go on.
*/
static inline bool result_should_goto_done(cli_ctx *ctx, cl_error_t result_in, cl_error_t *result_out)
{
bool halt_scan = false;
if (NULL == ctx || NULL == result_out) {
cli_dbgmsg("Invalid arguments for file scan result check.\n");
halt_scan = true;
goto done;
}
if (NULL != ctx && ctx->abort_scan) {
Do not report timeout as error Because of an oversight, the `ctx->abort_scan` flag was propagating timeout and perhaps other error types to the final verdict. Fixed this in the `result_should_goto_done()` function. Some non-fatal errors (notably CL_ETIMEOUT) may appear between the call to the magic-scan, and the end of scan_common(). We need to filter these status results at this level as well to prevent this from happening. I also fixed a warning accidentally introduced earlier caused by trying to use a string after free'ing it and setting it to NULL. And I explicitly set the status to CL_SUCCESS if the cache check comes back clean, rather than trusting that no one had messed with the `ret` variable since it was let set (to CL_SUCCESS). It's more readable and less prone to future bugs.
2022-09-21 12:22:54 -07:00
// ensure abort_scan is respected
halt_scan = true;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
switch (result_in) {
/*
* Reasons to halt the scan and report the error up to the caller/user.
*/
// A virus result means we should halt the scan.
// We do not return CL_VIRUS in allmatch-mode until the very end.
case CL_VIRUS:
// Each of these error codes considered terminal and will halt the scan.
case CL_EUNLINK:
case CL_ESTAT:
case CL_ESEEK:
case CL_EWRITE:
case CL_EDUP:
case CL_ETMPFILE:
case CL_ETMPDIR:
case CL_EMEM:
cli_dbgmsg("Descriptor[%d]: halting after file scan because: %s\n", fmap_fd(ctx->fmap), cl_strerror(result_in));
halt_scan = true;
*result_out = result_in;
break;
/*
* Reasons to halt the scan but report a successful scan.
*/
Fix typos (no functional changes)
2023-11-26 15:01:19 -08:00
// Exceeding the time limit should definitely halt the scan.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// But unless the user enabled alert-exceeds-max, we don't want to complain about it.
case CL_ETIMEOUT:
// If the file was determined to be trusted, then we can stop scanning this layer. (Ex: EXE with a valid Authenticode sig.)
// Convert CL_VERIFIED to CL_SUCCESS because we don't want to propagate the CL_VERIFIED return code up to the caller.
// If we didn't, a trusted file could cause a larger archive containing non-trustworthy files to be trusted.
case CL_VERIFIED:
cli_dbgmsg("Descriptor[%d]: halting after file scan because: %s\n", fmap_fd(ctx->fmap), cl_strerror(result_in));
halt_scan = true;
*result_out = CL_SUCCESS;
break;
/*
* All other results must not halt the scan.
*/
// Nothing to do.
case CL_SUCCESS:
// Unless ctx->abort_scan was set, all these "MAX" conditions should finish scanning as much as is allowed.
// That is, the can may still be blocked from recursing into the next layer, or scanning new files or large files.
case CL_EMAXREC:
case CL_EMAXSIZE:
case CL_EMAXFILES:
// The following are explicitly listed here so you think twice before putting them in the scan-halt list, above.
// Malformed/truncated files could report as any of these three, and that's fine.
// See commit 087e7fc3fa923e5d6a6fd2efe8df852a36256b5b for additional details.
case CL_EFORMAT:
case CL_EPARSE:
case CL_EREAD:
case CL_EUNPACK:
default:
cli_dbgmsg("Descriptor[%d]: Continuing after file scan resulted with: %s\n",
fmap_fd(ctx->fmap), cl_strerror(result_in));
*result_out = CL_SUCCESS;
}
done:
return halt_scan;
}
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cl_error_t cli_magic_scan(cli_ctx *ctx, cli_file_t type)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cl_error_t status = CL_SUCCESS;
cl_error_t ret;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_error_t cache_check_result = CL_VIRUS;
cl_verdict_t verdict_at_this_level = CL_VERDICT_NOTHING_FOUND;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
bool cache_enabled = true;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cli_file_t dettype = CL_TYPE_ANY;
libclamav: Fix benign uninitialized variable used In the core `cli_magic_scan()` function, early termination of the scan before hashing the file, such as a return from the prescan-callback, could result in the hashed_size variable being uninitalized when calling into `cache_add()`. That's okay because we now skip caching of the md5 hash when it is NULL, hence "benign". This is just a minor code quality fix.
2022-03-30 09:11:48 -07:00
uint8_t typercg = 1;
Fix fmap handle_gets() page arithmetic The fixes to the fmap bounds for nested (duplicate) fmaps added recently introduced a subtle arithmetic bug that was detected by OSS-Fuzz: ```c scanat = m->nested_offset + *at % m->pgsz; ``` should have been: ```c scanat = (m->nested_offset + *at) % m->pgsz; ``` Without the parenthesis, `scanat` could be > `m->pgsz`, which would overflow in the subsequent `memchr()` call. See: - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40452 - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40455 This commit also tightens up some of the other bounds checks done with `CLI_ISCONTAINED()` macro so the check limits the bounds to the nested fmap and not the original map. In addition, I've added a `CLI_ISCONTAINED_0_TO()` macro that removes checks when the "bigger" buffer starts at offset 0. This should silence a bunch of (benign) warnings and medium severity Coverity issues. There is also a possible use of an uninitialized variable (`old_hook_lsig_matches`) in `cli_magic_scan()`. Finally, I also removed an unecessary NULL-check on `filebase` in `fmap_dup_to_file()` that Coverity was unhappy with.
2021-10-28 16:58:21 -07:00
bitset_t *old_hook_lsig_matches = NULL;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
const char *filetype;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("CRITICAL: engine == NULL\n");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_ENULLARG;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto early_ret;
libclamav: report error instead of crashing when engine is not compiled (bb#1416) git-svn: trunk@4882
2009-03-02 16:36:23 +00:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!(ctx->engine->dboptions & CL_DB_COMPILED)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
cli_errmsg("CRITICAL: engine not compiled\n");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_EMALFDB;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto early_ret;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (ctx->fmap->len <= 5) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_SUCCESS;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
cli_dbgmsg("cli_magic_scan: File is too small (%zu bytes), ignoring.\n", ctx->fmap->len);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto early_ret;
bb #5405: Convert hard-coded maximum file sizes to configurable values with defaults
2012-11-27 17:15:02 -05:00
}
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (cli_updatelimits(ctx, ctx->fmap->len) != CL_SUCCESS) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
emax_reached(ctx);
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_SUCCESS;
cli_dbgmsg("cli_magic_scan: returning %d %s (no post, no cache)\n", status, __AT__);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto early_ret;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (type == CL_TYPE_PART_ANY) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
typercg = 0;
bb #1570: partition typing and HFS+
2013-09-17 16:45:48 -04:00
}
fmap in magicscan
2009-08-30 19:14:49 +02:00
libclamav: Fix issue scanning with no signatures loaded (#1560) If the application never calls `cl_load()`, then the clean-cache is never initialized. That is a legitimate mode to run in when perhaps we just want to extract stuff, record metadata, or for fuzzing. This commit adds a check if the ctx->engine->cache is NULL. If it is, then we treat it as though caching is disabled. CLAM-2856
2025-08-28 10:31:35 -04:00
/*
* Determine if caching is enabled.
* The application may have specifically disabled caching. Also, if the application never loaded any signatures,
* then the cache will be NULL and caching will also be disabled.
*/
if ((ctx->engine->engine_options & ENGINE_OPTIONS_DISABLE_CACHE) ||
(ctx->engine->cache == NULL)) {
Defer or avoid file MD5 calculation when cache is disabled
2022-11-18 23:48:40 -07:00
cache_enabled = false;
}
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* Perform file typing from the start of the file.
*/
rework scan callbacks
2011-06-14 17:00:06 +02:00
perf_start(ctx, PERFT_FT);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if ((type == CL_TYPE_ANY) || type == CL_TYPE_PART_ANY) {
ZIP: Fix NULL-dereference for OOXML scans (#1552) I accidentally introduced a NULL-dereference bug when scanning any OOXML file in https://github.com/Cisco-Talos/clamav/pull/1548 I overlooked the test failure out of haste. 😔 The NULL-dereference happens because the `unzip_search()` feature allowed searching some other file than the one that is currently being scanned, which you would do by setting `ctx` to NULL and setting an `fmap` parameter instead. In practice, the current layer's `fmap` from the `ctx` was always passed in. This fix makes it so the `unzip_search()` and related functions only take the `ctx` parameter and do not have and `fmap` or `fsize` field (Note: the `fsize` was never needed, because `fmap->len` take care of that). CLAM-2837
2025-08-14 21:17:46 -04:00
type = cli_determine_fmap_type(ctx, type);
mbr: lowered magic otf scanning priority scanners: removed redundant call to scan function pe_icons: collected 'broken image' dbgmsgs into single dbgmsg
2014-03-24 18:45:29 -04:00
}
rework scan callbacks
2011-06-14 17:00:06 +02:00
perf_stop(ctx, PERFT_FT);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (type == CL_TYPE_ERROR) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_EREAD;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
cli_dbgmsg("cli_magic_scan: cli_determine_fmap_type returned CL_TYPE_ERROR\n");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cli_dbgmsg("cli_magic_scan: returning %d %s (no post, no cache)\n", status, __AT__);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto early_ret;
add filetype clcb
2011-06-14 03:26:30 +02:00
}
rework scan callbacks
2011-06-14 17:00:06 +02:00
filetype = cli_ftname(type);
Add build support for libjson and initial json work for properties file.
2014-04-16 16:40:56 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
/* set current layer to the type we found */
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
ret = cli_recursion_stack_change_type(ctx, type, true /* ? */);
if (CL_SUCCESS != ret) {
cli_dbgmsg("cli_magic_scan: cli_recursion_stack_change_type returned %d\n", ret);
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
// We must go to done here (and not early_ret), because `ret` needs to be tidied up before returning.
status = ret;
goto done;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
* Run the pre_hash callback.
*/
ret = cli_dispatch_scan_callback(ctx, CL_SCAN_CALLBACK_PRE_HASH);
if (CL_SUCCESS != ret) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = ret;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
goto done;
}
/*
* Run the deprecated pre_cache callback.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
*/
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
ret = dispatch_prescan_callback(ctx->engine->cb_pre_cache, ctx, filetype, true /* pre_cache */);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_VERIFIED == ret || CL_VIRUS == ret) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = ret;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
goto done;
Convert two macros to functions
2014-04-30 15:01:05 -04:00
}
add filetype clcb
2011-06-14 03:26:30 +02:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
/*
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
* Run the deprecated file_inspection callback.
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
*/
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
ret = dispatch_file_inspection_callback(ctx->engine->cb_file_inspection, ctx, filetype);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
if (CL_SUCCESS != ret) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = ret;
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
goto done;
}
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
/*
* Record the file hash(es) in the JSON metadata before we do the cache check.
*/
if (SCAN_COLLECT_METADATA) {
uint8_t *hash = NULL;
char hash_string[SHA256_HASH_SIZE * 2 + 1];
bool need_hash[CLI_HASH_AVAIL_TYPES] = {false};
cli_hash_type_t hash_type;
need_hash[CLI_HASH_SHA2_256] = true;
if (SCAN_COLLECT_METADATA && SCAN_STORE_EXTRA_HASHES) {
need_hash[CLI_HASH_MD5] = true;
need_hash[CLI_HASH_SHA1] = true;
}
/* Set fmap to need hash later if required.
* This is an optimization so we can calculate all needed hashes in one pass. */
for (hash_type = CLI_HASH_MD5; hash_type < CLI_HASH_AVAIL_TYPES; hash_type++) {
if (need_hash[hash_type]) {
ret = fmap_will_need_hash_later(ctx->fmap, hash_type);
if (CL_SUCCESS != ret) {
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
cli_dbgmsg("cli_magic_scan: Failed to set fmap to need the %s hash later\n", cli_hash_name(hash_type));
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = ret;
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
goto done;
}
}
}
for (hash_type = CLI_HASH_MD5; hash_type < CLI_HASH_AVAIL_TYPES; hash_type++) {
if (need_hash[hash_type]) {
size_t i;
size_t hash_len = cli_hash_len(hash_type);
/* If we need a hash, we will calculate it now */
ret = fmap_get_hash(ctx->fmap, &hash, hash_type);
if (CL_SUCCESS != ret || hash == NULL) {
cli_dbgmsg("cli_magic_scan: Failed to get a hash for the current fmap.\n");
// It may be that the file was truncated between the time we started the scan and the time we got the hash.
// Not a reason to print an error message.
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_SUCCESS;
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
goto done;
}
/* Convert hash to string */
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
for (i = 0; i < hash_len; i++) {
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
sprintf(hash_string + i * 2, "%02x", hash[i]);
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
}
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
hash_string[hash_len * 2] = 0;
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
ret = cli_jsonstr(ctx->this_layer_metadata_json, cli_hash_name(hash_type), hash_string);
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
if (ret != CL_SUCCESS) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cli_dbgmsg("cli_magic_scan: Failed to store the %s hash in the metadata JSON.\n", cli_hash_name(hash_type));
status = ret;
goto done;
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
}
}
}
}
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* Check if we've already scanned this file before.
*/
Defer or avoid file MD5 calculation when cache is disabled
2022-11-18 23:48:40 -07:00
if (cache_enabled) {
perf_start(ctx, PERFT_CACHE);
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
cache_check_result = clean_cache_check(ctx);
Defer or avoid file MD5 calculation when cache is disabled
2022-11-18 23:48:40 -07:00
perf_stop(ctx, PERFT_CACHE);
}
json_api: returns error codes or CL_SUCCESS w or w/o JSON doc/ppt/xls: propagates error codes to topmost lvl w/ premature termination
2014-04-24 14:22:00 -04:00
Defer or avoid file MD5 calculation when cache is disabled
2022-11-18 23:48:40 -07:00
if (cache_enabled && (cache_check_result != CL_VIRUS)) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_SUCCESS;
cli_dbgmsg("cli_magic_scan: returning %d %s (no post, no cache)\n", status, __AT__);
// We can go to early_ret here, because we know status is CL_SUCCESS, and we obviously add to the cache.
// This does mean, however, that we do not run the post-scan callback for layers that are cached.
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto early_ret;
fix clamav-config include, properly restore fmap - thx edwin for the spot!
2010-01-05 18:15:59 +01:00
}
bb#4669
2012-05-08 15:34:27 +02:00
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
/* Save off the hook_lsig_matches */
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
old_hook_lsig_matches = ctx->hook_lsig_matches;
Support PE hook bytecodes triggered by logical signature.
2010-01-19 16:38:12 +02:00
ctx->hook_lsig_matches = NULL;
cache final fixes
2010-01-15 03:00:15 +01:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
* Run the pre_scan callback.
*/
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
ret = cli_dispatch_scan_callback(ctx, CL_SCAN_CALLBACK_PRE_SCAN);
if (CL_SUCCESS != ret) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = ret;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
goto done;
}
/*
* Run the deprecated pre_scan callback.
*/
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
ret = dispatch_prescan_callback(ctx->engine->cb_pre_scan, ctx, filetype, false /* pre_cache */);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (CL_VERIFIED == ret || CL_VIRUS == ret) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = ret;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
goto done;
}
// If none of the scan options are enabled, then we can skip parsing and just do a raw pattern match.
// For this check, we don't care if the CL_SCAN_GENERAL_ALLMATCHES option is enabled, hence the `~`.
Fix recursion limit alert The previous commit broke alerting when exceeding the recursion limit because recursion tracking is so effective that by limiting the final layer of recursion to a scan of the fmap, we prevented it from ever hitting the recursion limit. This commit removes that restriction where it only does an fmap scan (aka "raw scan") of files that are at their limit so that we can actually hit the recursion limit and alert as intended. Also tidied up the cache_clean check so it checks the `fmap->dont_cache_flag` at the right point (before caching) instead of before setting the "CLEAN" verdict. Note: The `cache_clean` variable appears to be used to record the clean status so the `ret` variable can be re-used without losing the verdict. This is of course only required because the verdict is stored in the error enum. *cough* Also fixed a couple typos.
2021-10-25 19:41:36 -07:00
if (!((ctx->options->general & ~CL_SCAN_GENERAL_ALLMATCHES) || (ctx->options->parse) || (ctx->options->heuristic) || (ctx->options->mail) || (ctx->options->dev))) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = cli_scan_fmap(ctx, CL_TYPE_ANY, false, NULL, AC_SCAN_VIR, NULL);
Fix issue reporting all Heuristic/PUA matches in allmatch mode When potentially unwanted indicators are found, they are not reported right away, except in heuristic-precedence mode. At the end of the scan, we then report all of the heuristic/PUA alerts. In my initial overhaul of the allmatch mode, I introduced an issue where it is never reporting more than 1 heuristic/PUA alert. This commit fixes that by reporting all potentially-unwanted indicators at the end. Note that instead of using `cli_virus_found_cb()` which would report the top layer file descriptor as well in the callback, I'm reporting `-1` for the file descriptor instead, because we don't know when the alert was added. If it was at a deeper layer in the file, we would not longer have access to it at the end of the scan. Also removed excessive debug statements '... infected with ...' that use `cli_get_last_virus()` because it doesn't really reflect all of the matches in allmatch mode, because the match would've been reported previously (so it is excessive), and because I don't like the word 'infected'. :)
2022-08-16 19:13:05 -07:00
// It doesn't matter what was returned, always go to the end after this. Raw mode! No parsing files!
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto done;
initial rewrite of limits git-svn-id: file:///var/lib/svn/clamav-devel/branches/newlimits@3591 77e5149b-7576-45b1-b177-96237e5ba77b
2008-02-06 21:19:10 +00:00
}
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
// We already saved the hook_lsig_matches (above)
// The ctx one is NULL at present.
Support PE hook bytecodes triggered by logical signature.
2010-01-19 16:38:12 +02:00
ctx->hook_lsig_matches = cli_bitset_init();
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if (NULL == ctx->hook_lsig_matches) {
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
status = CL_EMEM;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto done;
Don't leak bitset in magicscan.
2010-03-05 19:56:43 +02:00
}
Support PE hook bytecodes triggered by logical signature.
2010-01-19 16:38:12 +02:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (type != CL_TYPE_IGNORED && ctx->engine->sdb) {
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* If self protection mechanism enabled, do the scanraw() scan first
* before extracting with a file type parser.
*/
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = scanraw(ctx, type, 0, &dettype);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Evaluate the result from the scan to see if it end the scan of this layer early,
// and to decid if we should propagate an error or not.
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (result_should_goto_done(ctx, ret, &status)) {
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
goto done;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
add support for signature based self-protection mode git-svn: trunk@1831
2006-01-25 12:11:31 +00:00
}
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* Run the file type parsers that we normally use before the raw scan.
*/
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
perf_nested_start(ctx, PERFT_CONTAINER, PERFT_SCAN);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
switch (type) {
case CL_TYPE_IGNORED:
break;
case CL_TYPE_HWP3:
if (SCAN_PARSE_HWP3 && (DCONF_DOC & DOC_CONF_HWP))
ret = cli_scanhwp3(ctx);
break;
case CL_TYPE_HWPOLE2:
if (SCAN_PARSE_OLE2 && (DCONF_ARCH & ARCH_CONF_OLE2))
ret = cli_scanhwpole2(ctx);
break;
case CL_TYPE_XML_WORD:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_MSXML))
ret = cli_scanmsxml(ctx);
break;
case CL_TYPE_XML_XL:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_MSXML))
ret = cli_scanmsxml(ctx);
break;
case CL_TYPE_XML_HWP:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_HWP))
ret = cli_scanhwpml(ctx);
break;
case CL_TYPE_XDP:
if (SCAN_PARSE_PDF && (DCONF_DOC & DOC_CONF_PDF))
ret = cli_scanxdp(ctx);
break;
case CL_TYPE_RAR:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
case CL_TYPE_RARSFX:
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (have_rar && SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_RAR))
ret = cli_scanrar(ctx);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
case CL_TYPE_EGG:
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
case CL_TYPE_EGGSFX:
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_EGG))
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
ret = cli_scanegg(ctx);
Added new EGG archive extraction feature, written from scratch based on ESTsoft's EGG archive specification. EGG extraction support includes deflate, bzip2, and lzma decompression. AZO (LZO?) decompression not yet supported. Solid archives not yet supported. Split archives may have some limited success. This commit also includes updates to autoconf iconv.m4 file enable detection of libiconv in alternative install locations.
2018-10-08 12:59:42 -04:00
break;
Support for extracting attachments from OneNote section files Includes rudimentary support for getting slices from FMap's and for interacting with libclamav's context structure. For now will use a Cisco-Talos org fork of the onenote_parser until the feature to read open a onenote section from a slice (instead of from a filepath) is added to the upstream.
2023-09-29 12:49:29 -07:00
case CL_TYPE_ONENOTE:
if (SCAN_PARSE_ONENOTE && (DCONF_ARCH & DOC_CONF_ONENOTE))
ret = scan_onenote(ctx);
break;
clam-format touchup
2024-04-13 14:39:19 -04:00
Add parser for ALZ archives
2024-01-24 12:03:16 -08:00
case CL_TYPE_ALZ:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ALZ)) {
ret = cli_scanalz(ctx);
}
break;
Support for extracting attachments from OneNote section files Includes rudimentary support for getting slices from FMap's and for interacting with libclamav's context structure. For now will use a Cisco-Talos org fork of the onenote_parser until the feature to read open a onenote section from a slice (instead of from a filepath) is added to the upstream.
2023-09-29 12:49:29 -07:00
Add LHA/LZH archive support File type magic signatures chosen based on the extensions supported by Rust delharc crate. See: https://docs.rs/delharc/latest/delharc/
2023-11-07 23:00:30 -05:00
case CL_TYPE_LHA_LZH:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_LHA_LZH))
ret = scan_lha_lzh(ctx);
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_OOXML_WORD:
case CL_TYPE_OOXML_PPT:
case CL_TYPE_OOXML_XL:
case CL_TYPE_OOXML_HWP:
if (SCAN_PARSE_XMLDOCS && (DCONF_DOC & DOC_CONF_OOXML)) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (SCAN_COLLECT_METADATA && (ctx->this_layer_metadata_json != NULL)) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ret = cli_process_ooxml(ctx, type);
if (ret == CL_EMEM || ret == CL_ENULLARG) {
/* critical error */
break;
} else if (ret != CL_SUCCESS) {
/*
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
* non-critical return => allow for the CL_TYPE_ZIP scan to occur
* cli_process_ooxml other possible returns:
* CL_ETIMEOUT, CL_EMAXSIZE, CL_EMAXFILES, CL_EPARSE,
* CL_EFORMAT, CL_BREAK, CL_ESTAT
*/
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ret = CL_SUCCESS;
}
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
}
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
/* Extract the OOXML contents */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ZIP))
ret = cli_unzip(ctx);
break;
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
case CL_TYPE_ZIP:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ZIP)) {
if (ctx->recursion_stack[ctx->recursion_level].attributes & LAYER_ATTRIBUTES_EMBEDDED) {
/* If this is an embedded ZIP found by scanraw() with file type detection,
* then we only extract a single zip entry. */
ret = cli_unzip_single(ctx, 0);
} else {
ret = cli_unzip(ctx);
}
}
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_GZ:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_GZ))
ret = cli_scangzip(ctx);
break;
case CL_TYPE_BZ:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_BZ))
ret = cli_scanbzip(ctx);
break;
case CL_TYPE_XZ:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_XZ))
ret = cli_scanxz(ctx);
break;
case CL_TYPE_GPT:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_GPT))
ret = cli_scangpt(ctx, 0);
break;
case CL_TYPE_APM:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_APM))
ret = cli_scanapm(ctx);
break;
case CL_TYPE_ARJ:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ARJ))
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
ret = cli_scanarj(ctx);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
break;
case CL_TYPE_NULSFT:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_NSIS))
ret = cli_scannulsft(ctx, 0);
break;
case CL_TYPE_AUTOIT:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_AUTOIT))
ret = cli_scanautoit(ctx, 23);
break;
case CL_TYPE_MSSZDD:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_SZDD))
ret = cli_scanszdd(ctx);
break;
case CL_TYPE_MSCAB:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_CAB))
ret = cli_scanmscab(ctx, 0);
break;
case CL_TYPE_HTML:
if (SCAN_PARSE_HTML && (DCONF_DOC & DOC_CONF_HTML))
ret = cli_scanhtml(ctx);
break;
Scan performance improvements ClamAV's embedded file type recognition detects some files found in non-archive formats but for archive formats and compressed data streams like bzip2 and gzip, it will often detect file type magic bytes of compressed files and then attempt to parse the compressed data as if they were whole files, resulting in wasted CPU cycles and confusing warnings. This patch prevents embedded file type recognition for CL_TYPE_GZ and CL_TYPE_BZ. Also revert the UTF8 Byte Order Mark (BOM) detection and associated scanning of all text types as HTML files that had been added in 0.103. Scanning a file as HTML is not performant because it creates temp files and and normalizes the original files 3 ways. Better text type detection, transcoding, and HTML detection is probably still needed, but will have to wait. Scanning any embedded content that looked like text with the HTML parser impacts performance too much.
2020-12-17 19:28:15 -08:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_HTML_UTF16:
if (SCAN_PARSE_HTML && (DCONF_DOC & DOC_CONF_HTML))
ret = cli_scanhtml_utf16(ctx);
break;
case CL_TYPE_SCRIPT:
if ((DCONF_DOC & DOC_CONF_SCRIPT) && dettype != CL_TYPE_HTML)
ret = cli_scanscript(ctx);
break;
case CL_TYPE_SWF:
if (SCAN_PARSE_SWF && (DCONF_DOC & DOC_CONF_SWF))
ret = cli_scanswf(ctx);
break;
case CL_TYPE_RTF:
if (SCAN_PARSE_ARCHIVE && (DCONF_DOC & DOC_CONF_RTF))
ret = cli_scanrtf(ctx);
break;
case CL_TYPE_MAIL:
if (SCAN_PARSE_MAIL && (DCONF_MAIL & MAIL_CONF_MBOX))
ret = cli_scanmail(ctx);
break;
case CL_TYPE_MHTML:
if (SCAN_PARSE_MAIL && (DCONF_MAIL & MAIL_CONF_MBOX))
ret = cli_scanmail(ctx);
break;
case CL_TYPE_TNEF:
if (SCAN_PARSE_MAIL && (DCONF_MAIL & MAIL_CONF_TNEF))
ret = cli_scantnef(ctx);
break;
case CL_TYPE_UUENCODED:
if (DCONF_OTHER & OTHER_CONF_UUENC)
ret = cli_scanuuencoded(ctx);
break;
case CL_TYPE_MSCHM:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_CHM))
ret = cli_scanmschm(ctx);
break;
case CL_TYPE_MSOLE2:
if (SCAN_PARSE_OLE2 && (DCONF_ARCH & ARCH_CONF_OLE2))
ret = cli_scanole2(ctx);
break;
case CL_TYPE_7Z:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_7Z))
ret = cli_7unz(ctx, 0);
break;
case CL_TYPE_POSIX_TAR:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_TAR))
ret = cli_scantar(ctx, 1);
break;
case CL_TYPE_OLD_TAR:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_TAR))
ret = cli_scantar(ctx, 0);
break;
case CL_TYPE_CPIO_OLD:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_CPIO))
ret = cli_scancpio_old(ctx);
break;
case CL_TYPE_CPIO_ODC:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_CPIO))
ret = cli_scancpio_odc(ctx);
break;
case CL_TYPE_CPIO_NEWC:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_CPIO))
ret = cli_scancpio_newc(ctx, 0);
break;
case CL_TYPE_CPIO_CRC:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_CPIO))
ret = cli_scancpio_newc(ctx, 1);
break;
case CL_TYPE_BINHEX:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_BINHEX))
ret = cli_binhex(ctx);
break;
case CL_TYPE_SCRENC:
if (DCONF_OTHER & OTHER_CONF_SCRENC)
ret = cli_scanscrenc(ctx);
break;
case CL_TYPE_RIFF:
if (SCAN_HEURISTICS && (DCONF_OTHER & OTHER_CONF_RIFF))
ret = cli_scanriff(ctx);
break;
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
case CL_TYPE_GRAPHICS: {
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE) {
/*
* This case is for unhandled graphics types such as BMP, JPEG 2000, etc.
*
* Note: JPEG 2000 is a very different format from JPEG, JPEG/JFIF, JPEG/Exif, JPEG/SPIFF (1994, 1997)
* JPEG 2000 is not handled by cli_parsejpeg.
*/
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE_FUZZY_HASH && (DCONF_OTHER & OTHER_CONF_IMAGE_FUZZY_HASH)) {
// It's okay if it fails to calculate the fuzzy hash.
(void)calculate_fuzzy_image_hash(ctx, type);
}
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
break;
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
case CL_TYPE_GIF: {
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE && (DCONF_OTHER & OTHER_CONF_GIF)) {
if (SCAN_HEURISTICS && SCAN_HEURISTIC_BROKEN_MEDIA) {
/*
* Parse GIF files, checking for exploits and other file format issues.
*/
ret = cli_parsegif(ctx);
if (CL_SUCCESS != ret) {
// do not calculate the fuzzy image hash if parsing failed, or a heuristic alert occurred.
break;
}
}
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE_FUZZY_HASH && (DCONF_OTHER & OTHER_CONF_IMAGE_FUZZY_HASH)) {
// It's okay if it fails to calculate the fuzzy hash.
(void)calculate_fuzzy_image_hash(ctx, type);
}
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
gif - clang formatted; copyright dates fixed
2019-09-16 14:56:27 -04:00
break;
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
case CL_TYPE_PNG: {
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE && (DCONF_OTHER & OTHER_CONF_PNG)) {
if (SCAN_HEURISTICS && SCAN_HEURISTIC_BROKEN_MEDIA) {
/*
* Parse PNG files, checking for exploits and other file format issues.
*/
ret = cli_parsepng(ctx); /* PNG parser detects a couple CVE's as well as Broken.Media */
if (CL_SUCCESS != ret) {
// do not calculate the fuzzy image hash if parsing failed, or a heuristic alert occurred.
break;
}
}
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE_FUZZY_HASH && (DCONF_OTHER & OTHER_CONF_IMAGE_FUZZY_HASH)) {
// It's okay if it fails to calculate the fuzzy hash.
(void)calculate_fuzzy_image_hash(ctx, type);
}
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
GIF, PNG bugfixes; Add AlertBrokenMedia option Added a new scan option to alert on broken media (graphics) file formats. This feature mitigates the risk of malformed media files intended to exploit vulnerabilities in other software. At present media validation exists for JPEG, TIFF, PNG, and GIF files. To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or use the `--alert-broken-media` option when using `clamscan`. These options are disabled by default for now. Application developers may enable this scan option by enabling `CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit field. Fixed PNG parser logic bugs that caused an excess of parsing errors and fixed a stack exhaustion issue affecting some systems when scanning PNG files. PNG file type detection was disabled via signature database update for 0.103.0 to mitigate effects from these bugs. Fixed an issue where PNG and GIF files no longer work with Target:5 (graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as CL_TYPE_GRAPHICS. Target types now support up to 10 possible file types to make way for additional graphics types in future releases. Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse" errors when file format validation fails. Instead, the scan will alert with the "Heuristics.Broken.Media" signature prefix and a descriptive suffix to indicate the issue, provided that the "alert broken media" feature is enabled. GIF format validation will no longer fail if the GIF image is missing the trailer byte, as this appears to be a relatively common issue in otherwise functional GIF files. Added a TIFF dynamic configuration (DCONF) option, which was missing. This will allow us to disable TIFF format validation via signature database update in the event that it proves to be problematic. This feature already exists for many other file types. Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
2020-11-04 15:49:43 -08:00
break;
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
case CL_TYPE_JPEG: {
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE && (DCONF_OTHER & OTHER_CONF_JPEG)) {
if (SCAN_HEURISTICS && SCAN_HEURISTIC_BROKEN_MEDIA) {
/*
* Parse JPEG files, checking for exploits and other file format issues.
*
* Note: JPEG 2000 is a very different format from JPEG, JPEG/JFIF, JPEG/Exif, JPEG/SPIFF (1994, 1997)
* JPEG 2000 is not checked by cli_parsejpeg.
*/
ret = cli_parsejpeg(ctx); /* JPG parser detects MS04-028 exploits as well as Broken.Media */
if (CL_SUCCESS != ret) {
// do not calculate the fuzzy image hash if parsing failed, or a heuristic alert occurred.
break;
}
}
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE_FUZZY_HASH && (DCONF_OTHER & OTHER_CONF_IMAGE_FUZZY_HASH)) {
// It's okay if it fails to calculate the fuzzy hash.
(void)calculate_fuzzy_image_hash(ctx, type);
}
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
GIF, PNG bugfixes; Add AlertBrokenMedia option Added a new scan option to alert on broken media (graphics) file formats. This feature mitigates the risk of malformed media files intended to exploit vulnerabilities in other software. At present media validation exists for JPEG, TIFF, PNG, and GIF files. To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or use the `--alert-broken-media` option when using `clamscan`. These options are disabled by default for now. Application developers may enable this scan option by enabling `CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit field. Fixed PNG parser logic bugs that caused an excess of parsing errors and fixed a stack exhaustion issue affecting some systems when scanning PNG files. PNG file type detection was disabled via signature database update for 0.103.0 to mitigate effects from these bugs. Fixed an issue where PNG and GIF files no longer work with Target:5 (graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as CL_TYPE_GRAPHICS. Target types now support up to 10 possible file types to make way for additional graphics types in future releases. Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse" errors when file format validation fails. Instead, the scan will alert with the "Heuristics.Broken.Media" signature prefix and a descriptive suffix to indicate the issue, provided that the "alert broken media" feature is enabled. GIF format validation will no longer fail if the GIF image is missing the trailer byte, as this appears to be a relatively common issue in otherwise functional GIF files. Added a TIFF dynamic configuration (DCONF) option, which was missing. This will allow us to disable TIFF format validation via signature database update in the event that it proves to be problematic. This feature already exists for many other file types. Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
2020-11-04 15:49:43 -08:00
break;
Image fuzzy hash: new logical sub-signature feature Add a new logical signature subsignature type for matching on images with image fuzzy hashes. Image fuzzy hash subsigantures follow this format: fuzzy_img#<hash>#<dist> In this initial implementation, the hamming distance (dist) is ignored and only exact fuzzy hash matches will alert. Fuzzy hash matching is only performed for supported image types. Also: removed some excessive debug log messages on start-up. Fixed an issue where the signature name (virname) is being allocated and stored for every subsignature or even ever sub-pattern in an AC-pattern (i.e. NDB sig or LDB subsig) containing a `{n-m}` or `*` wildcard. This fix is only for LDB subsigs though. NDB signatures are still allocaing one virname per sub-pattern. This fix was required because I needed a place to store the virname with fuzzy-hash subsignatures. Storing it in the fuzzy-hash subsig metadatathe way AC-pattern, PCRE, and BComp subsigs were doing it wouldn't work because it would cross the C-Rust FFI boundary and giving pointers to Rust allocated stuff is dicey. Not to mention native Rust strings are different thatn C strings. Anyways, the correct thing to do was to store the virname with the actual logical signature. TODO: Keep track of NDB signatures in the same way and store the virname for NDB sigs there instead of in AC-patterns so that we can get rid of the virname field in the AC-pattern struct.
2021-04-21 16:24:24 -07:00
}
GIF, PNG bugfixes; Add AlertBrokenMedia option Added a new scan option to alert on broken media (graphics) file formats. This feature mitigates the risk of malformed media files intended to exploit vulnerabilities in other software. At present media validation exists for JPEG, TIFF, PNG, and GIF files. To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or use the `--alert-broken-media` option when using `clamscan`. These options are disabled by default for now. Application developers may enable this scan option by enabling `CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit field. Fixed PNG parser logic bugs that caused an excess of parsing errors and fixed a stack exhaustion issue affecting some systems when scanning PNG files. PNG file type detection was disabled via signature database update for 0.103.0 to mitigate effects from these bugs. Fixed an issue where PNG and GIF files no longer work with Target:5 (graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as CL_TYPE_GRAPHICS. Target types now support up to 10 possible file types to make way for additional graphics types in future releases. Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse" errors when file format validation fails. Instead, the scan will alert with the "Heuristics.Broken.Media" signature prefix and a descriptive suffix to indicate the issue, provided that the "alert broken media" feature is enabled. GIF format validation will no longer fail if the GIF image is missing the trailer byte, as this appears to be a relatively common issue in otherwise functional GIF files. Added a TIFF dynamic configuration (DCONF) option, which was missing. This will allow us to disable TIFF format validation via signature database update in the event that it proves to be problematic. This feature already exists for many other file types. Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
2020-11-04 15:49:43 -08:00
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
case CL_TYPE_TIFF: {
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE && (DCONF_OTHER & OTHER_CONF_TIFF)) {
if (SCAN_HEURISTICS && SCAN_HEURISTIC_BROKEN_MEDIA) {
/*
* Parse TIFF files, checking for exploits and other file format issues.
*/
ret = cli_parsetiff(ctx);
if (CL_SUCCESS != ret) {
// do not calculate the fuzzy image hash if parsing failed, or a heuristic alert occurred.
break;
}
}
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
Make image fuzzy hashing optional Image fuzzy hashing is enabled by default. The following options have been added to allow users to disable it, if desired. New clamscan options: --scan-image[=yes(*)/no] --scan-image-fuzzy-hash[=yes(*)/no] New clamd config options: ScanImage yes(*)/no ScanImageFuzzyHash yes(*)/no New libclamav scan options: options.parse &= ~CL_SCAN_PARSE_IMAGE; options.parse &= ~CL_SCAN_PARSE_IMAGE_FUZZY_HASH; This commit also changes scan behavior to disable image fuzzy hashing for specific types when the DCONF (.cfg) signatures disable those types. That is, if DCONF disables the PNG parser, it should not only disable the CVE/format checker for PNG files, but also disable image fuzzy hashing for PNG files. Also adds a DCONF option to disable image fuzzy hashing: OTHER_CONF_IMAGE_FUZZY_HASH DCONF allows scanning features to be disabled using a configuration "signature".
2024-02-25 12:43:56 -05:00
if (SCAN_PARSE_IMAGE_FUZZY_HASH && (DCONF_OTHER & OTHER_CONF_IMAGE_FUZZY_HASH)) {
// It's okay if it fails to calculate the fuzzy hash.
(void)calculate_fuzzy_image_hash(ctx, type);
}
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
}
Transforming the PNG checker into a PNG exploit seeker
2017-02-21 19:58:09 +01:00
break;
Fix scan error when image fuzzy hash fails Scanning a file containing an image where it fails to calculate the image fuzzy hash may cause the entire scan to end with an error. This commit fixes that by ignoring any image fuzzy hash failure. If calculating the hash fails, the hash is left uninitialized (all zeroes). To prevent doing image fuzzy hash lookups when it failed, I added a boolean that is set to true only after creating the hash. Fixes: https://github.com/Cisco-Talos/clamav/issues/600 Fixes: https://github.com/Cisco-Talos/clamav/issues/593
2022-06-21 16:43:39 -07:00
}
Transforming the PNG checker into a PNG exploit seeker
2017-02-21 19:58:09 +01:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_CRYPTFF:
if (DCONF_OTHER & OTHER_CONF_CRYPTFF)
ret = cli_scancryptff(ctx);
break;
case CL_TYPE_ELF:
if (SCAN_PARSE_ELF && ctx->dconf->elf)
ret = cli_scanelf(ctx);
break;
case CL_TYPE_MACHO:
if (ctx->dconf->macho)
ret = cli_scanmacho(ctx, NULL);
break;
case CL_TYPE_MACHO_UNIBIN:
if (ctx->dconf->macho)
ret = cli_scanmacho_unibin(ctx);
break;
case CL_TYPE_SIS:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_SIS))
ret = cli_scansis(ctx);
break;
case CL_TYPE_XAR:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_XAR))
ret = cli_scanxar(ctx);
break;
case CL_TYPE_PART_HFSPLUS:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_HFSPLUS))
ret = cli_scanhfsplus(ctx);
break;
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
case CL_TYPE_ISHIELD_MSI:
if (SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ISHIELD))
ret = cli_scanishield_msi(ctx, 14);
break;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_BINARY_DATA:
case CL_TYPE_TEXT_UTF16BE:
if (SCAN_HEURISTICS && (DCONF_OTHER & OTHER_CONF_MYDOOMLOG))
ret = cli_check_mydoom_log(ctx);
break;
case CL_TYPE_TEXT_ASCII:
if (SCAN_HEURISTIC_STRUCTURED && (DCONF_OTHER & OTHER_CONF_DLP))
/* TODO: consider calling this from cli_scanscript() for
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
* a normalised text
*/
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ret = cli_scan_structured(ctx);
break;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
default:
break;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
perf_nested_stop(ctx, PERFT_CONTAINER, PERFT_SCAN);
update git-svn: trunk@515
2004-04-20 22:33:42 +00:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Evaluate the result from the parsers to see if it end the scan of this layer early,
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
// and to decide if we should propagate an error or not.
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (result_should_goto_done(ctx, ret, &status)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
goto done;
}
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* Perform the raw scan, which may include file type recognition signatures.
*/
integrate the DLP code from Marty Roesch git-svn: trunk@3795
2008-04-16 18:47:42 +00:00
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/* Disable type recognition for the raw scan for zip files larger than maxziptypercg */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (type == CL_TYPE_ZIP && SCAN_PARSE_ARCHIVE && (DCONF_ARCH & ARCH_CONF_ZIP)) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
/* CL_ENGINE_MAX_ZIPTYPERCG */
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
uint64_t curr_len = ctx->fmap->len;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (curr_len > ctx->engine->maxziptypercg) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_dbgmsg("cli_magic_scan: Not checking for embedded PEs (zip file > MaxZipTypeRcg)\n");
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
typercg = 0;
}
don't search for embedded PEs in zip files larger than 1 MB (bb#573) git-svn: trunk@3155
2007-07-16 15:58:54 +00:00
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
/*
* Perform pattern matching for malware detections AND embedded file type recognition.
* Embedded file type recognition may re-assign the current file as a new type, or
* it may detect embedded files. E.g. ZIP entries in a PE file (i.e. self-extracting ZIP).
*/
if ((type != CL_TYPE_IGNORED) &&
/* CL_TYPE_HTML: raw HTML files are not scanned, unless safety measure activated via DCONF */
(type != CL_TYPE_HTML || !(SCAN_PARSE_HTML) || !(DCONF_DOC & DOC_CONF_HTML_SKIPRAW)) &&
(!ctx->engine->sdb)) {
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = scanraw(ctx, type, typercg, &dettype);
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Evaluate the result from the scan to see if it end the scan of this layer early,
// and to decid if we should propagate an error or not.
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (result_should_goto_done(ctx, ret, &status)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
goto done;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
GPT parser verbosity, DMG/BZ2 detection fix Reduced the verbosity of a GPT parser warning that occurs frequently when parsing DMG files prior to DMG file type recognition. DMG files support a handful of compression formats. File type recognition for DMG presently works by doing "embedded" file type recognition during the raw scan after having already identified the file type by traditional file type magic checks. I found that when DMG uses bzip2 for compression, we identify an MBR type containing a BZ type, at which point the raw scan detects it as DMG. The previous commits broke this by disabling embedded file type recognition for BZ and other compression & archivie types. Ideally the fix would be to do DMG file type detection by checking the end of the file; perhaps adding negative offset support for FTM sigs could fix it. Until we can implement that or another/better solution for DMG file type detection, we'll have to allow embedded file type recognition for BZ files. Also added some comments to narrate the scan process.
2021-01-19 14:23:02 -08:00
/*
* Now run the rest of the file type parsers.
*/
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
switch (type) {
/* bytecode hooks triggered by a lsig must be a hook
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
* called from one of the functions here */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_TEXT_ASCII:
case CL_TYPE_TEXT_UTF16BE:
case CL_TYPE_TEXT_UTF16LE:
case CL_TYPE_TEXT_UTF8:
perf_nested_start(ctx, PERFT_SCRIPT, PERFT_SCAN);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
if ((dettype != CL_TYPE_HTML) &&
SCAN_PARSE_HTML && (DCONF_DOC & DOC_CONF_SCRIPT) && (ret != CL_VIRUS)) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ret = cli_scanscript(ctx);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
if (((dettype == CL_TYPE_MAIL) || (cli_recursion_stack_get_type(ctx, -1) == CL_TYPE_MAIL)) &&
SCAN_PARSE_MAIL && (DCONF_MAIL & MAIL_CONF_MBOX) && (ret != CL_VIRUS)) {
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_scan_fmap(ctx, CL_TYPE_MAIL, false, NULL, AC_SCAN_VIR, NULL);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
perf_nested_stop(ctx, PERFT_SCRIPT, PERFT_SCAN);
break;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
/* Due to performance reasons all executables were first scanned
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
* in raw mode. Now we will try to unpack them
*/
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_MSEXE:
perf_nested_start(ctx, PERFT_PE, PERFT_SCAN);
if (SCAN_PARSE_PE && ctx->dconf->pe) {
Always run callbacks for embedded files Scans containing embedded files (e.g. appended ZIP archives) are not running the callback functions (pre-scan, pre-cache, inspection, etc.) on the embedded files. Run the cli_magic_scan() function for every file we extract using embedded file type recognition. CLAM-2796
2025-06-17 20:08:50 -04:00
if (ctx->recursion_stack[ctx->recursion_level].attributes & LAYER_ATTRIBUTES_EMBEDDED) {
/*
* Embedded PE files are PE files that were found within another file using file-type scanning in scanraw()
* They are parsed differently than normal PE files.
*/
struct cli_exe_info peinfo;
cli_exe_info_init(&peinfo, 0);
// TODO We could probably substitute in a quicker
// method of determining whether a PE file exists
// at this offset.
if (cli_peheader(ctx->fmap, &peinfo, CLI_PEHEADER_OPT_NONE, NULL) != 0) {
cli_dbgmsg("Header check for MSEXE detection failed, probably not actually an embedded PE file.\n");
/* Despite failing, peinfo memory may have been allocated and must be freed. */
cli_exe_info_destroy(&peinfo);
} else {
/* Immediately free up peinfo allocated memory, prior to any recursion */
cli_exe_info_destroy(&peinfo);
ret = cli_scanembpe(ctx, 0);
// TODO This method of embedded PE extraction
// is kinda gross in that:
// - if you have an executable that contains
// 20 other exes, the bytes associated with
// the last exe will have been included in
// hash computations and things 20 times
// (as overlay data to the previously
// extracted exes).
// - if you have a signed embedded exe, it
// will fail to validate after extraction
// bc it has overlay data, which is a
// violation of the Authenticode spec.
// - this method of extraction is subject to
// the recursion limit, which is fairly low.
//
// It'd be awesome if we could compute the PE
// size from the PE header and just extract
// that.
}
} else {
// Setting ctx->corrupted_input will prevent the PE parser from reporting "broken executable" for unpacked/reconstructed files that may not be 100% to spec.
// In here we're just carrying the corrupted_input flag from parent to child, in case the parent's flag was set.
unsigned int corrupted_input = ctx->corrupted_input;
ret = cli_scanpe(ctx);
ctx->corrupted_input = corrupted_input;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
}
perf_nested_stop(ctx, PERFT_PE, PERFT_SCAN);
break;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Linux bytecode unpackers
2019-03-19 15:28:49 +01:00
case CL_TYPE_ELF:
perf_nested_start(ctx, PERFT_ELF, PERFT_SCAN);
ret = cli_unpackelf(ctx);
perf_nested_stop(ctx, PERFT_ELF, PERFT_SCAN);
break;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Mach-O bytecode unpackers
2019-04-17 20:30:21 +02:00
case CL_TYPE_MACHO:
case CL_TYPE_MACHO_UNIBIN:
perf_nested_start(ctx, PERFT_MACHO, PERFT_SCAN);
ret = cli_unpackmacho(ctx);
perf_nested_stop(ctx, PERFT_MACHO, PERFT_SCAN);
break;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Add CL_TYPE_AI_MODEL and associated file type magic signatures This is just preliminary support for identifying an assortment of different AI model files. So far, this detects the following types: - GGML GGUF (.gguf) - ONNX AI (.onnx) - TensorFlow Lite (.tflite) Additional types to consider: - SafeTensors (.safetensors) - TensorFlow (.pb, .ckpt, .tfrecords) - Keras (.keras) - pickle (.pkl) - numpy (.npy, .npz) - coreml (.coreml) - PyTorch (.pt, .pth, .bin, .mar, .pte, .pt2, .ptl) Outside of being able to differentiate by file type, the scanner will treat CL_TYPE_AI_MODEL the same as CL_TYPE_BINARY_DATA. We're not adding parsers to further process these files, for now.
2025-03-27 14:15:48 -04:00
case CL_TYPE_AI_MODEL:
Add CL_TYPE_PYTHON_COMPILED and associated file type magic signatures It may be necessary to differentiate between *.pyc and other binary types in case additional processing is needed. Outside of being able to differentiate the by file type, the scanner will treat CL_TYPE_PYTHON_COMPILED the same as CL_TYPE_BINARY_DATA. That is - we're not adding parser at this time to further break down .pyc files.
2023-12-12 22:24:03 -05:00
case CL_TYPE_PYTHON_COMPILED:
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
case CL_TYPE_BINARY_DATA:
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
ret = cli_scan_fmap(ctx, CL_TYPE_OTHER, false, NULL, AC_SCAN_VIR, NULL);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
break;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
Fix bytecode match evaluation for PDF scans Bytecode signatures targeting PDF files (target 10) fail to evaluate match conditions. This occurs because the raw scan step during a magic scan currently occurs _after_ the PDF scan, thus at the time the hook is triggered, no matches have been performed and the logical sig eval is always negative. This commit fixes the issue by relocating the PDF parsing step to occur after the raw scan, thereby enabling the bytecode lsig evaluation to match and the signature to execute.
2020-03-19 21:15:56 -04:00
case CL_TYPE_PDF: /* FIXMELIMITS: pdf should be an archive! */
if (SCAN_PARSE_PDF && (DCONF_DOC & DOC_CONF_PDF))
ret = cli_scanpdf(ctx, 0);
break;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
default:
break;
scan raw executables before trying to unpack them git-svn: trunk@665
2004-07-11 14:50:25 +00:00
}
call checkfp before unmapping - just in case
2010-02-15 02:02:54 +01:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
done:
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
/*
* Run the post_scan callback.
*/
ret = cli_dispatch_scan_callback(ctx, CL_SCAN_CALLBACK_POST_SCAN);
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (CL_SUCCESS != ret) {
cli_dbgmsg("cli_magic_scan: POST_SCAN callback returned %d\n", ret);
status = ret;
}
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Filter the result from the parsers so we don't propagate non-fatal errors.
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
// And to convert CL_VERIFIED -> CL_SUCCESS
(void)result_should_goto_done(ctx, status, &status);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
* Run the deprecated post-scan callback (if one exists) and provide the verdict for this layer.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
*/
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cli_dbgmsg("cli_magic_scan: returning %d %s\n", status, __AT__);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
if (ctx->engine->cb_post_scan) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cl_error_t callback_ret;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
cl_error_t append_ret;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
const char *virusname = NULL;
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
// Get the last signature that matched (if any).
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (0 < evidence_num_alerts(ctx->this_layer_evidence)) {
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
virusname = cli_get_last_virus(ctx);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
perf_start(ctx, PERFT_POSTCB);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
callback_ret = ctx->engine->cb_post_scan(fmap_fd(ctx->fmap), verdict_at_this_level, virusname, ctx->cb_ctx);
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
perf_stop(ctx, PERFT_POSTCB);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
switch (callback_ret) {
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
case CL_BREAK:
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_dbgmsg("cli_magic_scan: file allowed by post_scan callback\n");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
// Remove any evidence for this layer and set the verdict to trusted.
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
(void)cli_trust_this_layer(ctx, "legacy post-scan application callback");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
// status = CL_SUCCESS; // Do override the status here.
// If status == CL_VIRUS, we'll fix when we look at the verdict.
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
break;
case CL_VIRUS:
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_dbgmsg("cli_magic_scan: file blocked by post_scan callback\n");
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
append_ret = cli_append_virus(ctx, "Detected.By.Callback");
if (append_ret == CL_VIRUS) {
status = CL_VIRUS;
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
break;
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
case CL_SUCCESS:
// No action requested by callback. Keep scanning.
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
break;
default:
Metadata JSON: Simplify recording alerts and indicators We presently record Alerts as an array of signature names. Instead, it should be an object with properties of its own. We should record alerting indicators and weak indicators in a single "Indicators", likely with the same structure as the "Alerts" objects. When an alerting indicator is ignored (e.g. ignored by callback or if the file is trusted by an FP signature), we can remove it from the "Alerts" array, and for the "Indicators" array, add a "Ignored" key with a string value that explains why it was ignored. This eliminates the need to track and propagate the additional "WeakIndicators" and "IgnoredAlerts" arrays.
2025-08-10 20:01:55 -04:00
// status = CL_SUCCESS; // Do override the status here, just log a warning.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_warnmsg("cli_magic_scan: ignoring bad return code from post_scan callback\n");
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
}
Fix recursion limit alert The previous commit broke alerting when exceeding the recursion limit because recursion tracking is so effective that by limiting the final layer of recursion to a scan of the fmap, we prevented it from ever hitting the recursion limit. This commit removes that restriction where it only does an fmap scan (aka "raw scan") of files that are at their limit so that we can actually hit the recursion limit and alert as intended. Also tidied up the cache_clean check so it checks the `fmap->dont_cache_flag` at the right point (before caching) instead of before setting the "CLEAN" verdict. Note: The `cache_clean` variable appears to be used to record the clean status so the `ret` variable can be re-used without losing the verdict. This is of course only required because the verdict is stored in the error enum. *cough* Also fixed a couple typos.
2021-10-25 19:41:36 -07:00
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
/*
* Check the verdict for this layer.
* If the verdict is CL_VERDICT_TRUSTED, remove any evidence for this layer and clear CL_VIRUS status (if set)
* Otherwise, we'll update the verdict based on the evidence.
*/
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (CL_VERDICT_TRUSTED == ctx->recursion_stack[ctx->recursion_level].verdict) {
/* Remove any alerts for this layer. */
if (NULL != ctx->recursion_stack[ctx->recursion_level].evidence) {
evidence_free(ctx->recursion_stack[ctx->recursion_level].evidence);
ctx->recursion_stack[ctx->recursion_level].evidence = NULL;
ctx->this_layer_evidence = NULL;
}
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
if (CL_VIRUS == status) {
status = CL_SUCCESS; // If we have a CL_VERDICT_TRUSTED, we should not return CL_VIRUS.
}
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
} else {
/*
* Update the verdict for this layer based on the scan results.
* If the verdict is CL_VERDICT_TRUSTED, then we don't change it.
*/
if (0 < evidence_num_indicators_type(ctx->this_layer_evidence, IndicatorType_Strong)) {
ctx->recursion_stack[ctx->recursion_level].verdict = CL_VERDICT_STRONG_INDICATOR;
} else if (0 < evidence_num_indicators_type(ctx->this_layer_evidence, IndicatorType_PotentiallyUnwanted)) {
ctx->recursion_stack[ctx->recursion_level].verdict = CL_VERDICT_POTENTIALLY_UNWANTED;
}
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
* If the verdict for this layer is "clean", we can cache it.
Swap clean cache from MD5 to SHA2-256 Change the clean-cache to use SHA2-256 instead of MD5. Note that all references are changed to specify "SHA2-256" now instead of "SHA256", for clarity. But there is no plan to add support for SHA3 algorithms at this time. Significant code cleanup. E.g.: - Implemented goto-done error handling. - Used `uint8_t *` instead of `unsigned char *`. - Use `bool` for boolean checks, rather than `int. - Used `#defines` instead of magic numbers. - Removed duplicate `#defines` for things like hash length. Add new option to calculate and record additional hash types when the "generate metadata JSON" feature is enabled: - libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES` - clamscan option: `--json-store-extra-hashes` (default off) - clamd.conf option: `JsonStoreExtraHashes` (default 'no') Renamed the sigtool option `--sha256` to `--sha2-256`. The original option is still functional, but is deprecated. For the "generate metadata JSON" feature, the file hash is now stored as "sha2-256" instead of "FileMD5". If you enable the "extra hashes" option, then it will also record "md5" and "sha1". Deprecate and disable the internal "SHA collect" feature. This option had been hidden behind C #ifdef checks for an option that wasn't exposed through CMake, so it was basically unavailable anyways. Changes to calculate file hashes when they're needed and no sooner. For the FP feature in the matcher module, I have mimiced the optimization in the FMAP scan routine which makes it so that it can calculate multiple hashes in a single pass of the file. The `HandlerType` feature stores a hash of the file in the scan ctx to prevent retyping the exact same data more than once. I removed that hash field and replaced it with an attribute flag that is applied to the new recursion stack layer when retyping a file. This also closes a minor bug that would prevent retyping a file with an all-zero hash. :) The work upgrading cache.c to support SHA2-256 sized hashes thanks to: https://github.com/m-sola CLAM-255 CLAM-1858 CLAM-1859 CLAM-1860
2025-06-03 19:03:20 -04:00
*
* Note: clean_cache_add() will check the fmap->dont_cache_flag,
* so this may not actually cache if we exceeded limits earlier.
* It will also check if caching is disabled.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
*/
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if ((CL_VERDICT_TRUSTED == ctx->recursion_stack[ctx->recursion_level].verdict) ||
(CL_VERDICT_NOTHING_FOUND == ctx->recursion_stack[ctx->recursion_level].verdict)) {
// Also verify we have no weak indicators before adding to the clean cache.
// Weak indicators may be used in the future to match a strong indicator.
if (evidence_num_indicators_type(ctx->this_layer_evidence, IndicatorType_Weak) == 0) {
perf_start(ctx, PERFT_CACHE);
clean_cache_add(ctx);
perf_stop(ctx, PERFT_CACHE);
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
Fix recursion limit alert The previous commit broke alerting when exceeding the recursion limit because recursion tracking is so effective that by limiting the final layer of recursion to a scan of the fmap, we prevented it from ever hitting the recursion limit. This commit removes that restriction where it only does an fmap scan (aka "raw scan") of files that are at their limit so that we can actually hit the recursion limit and alert as intended. Also tidied up the cache_clean check so it checks the `fmap->dont_cache_flag` at the right point (before caching) instead of before setting the "CLEAN" verdict. Note: The `cache_clean` variable appears to be used to record the clean status so the `ret` variable can be re-used without losing the verdict. This is of course only required because the verdict is stored in the error enum. *cough* Also fixed a couple typos.
2021-10-25 19:41:36 -07:00
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
early_ret:
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
if (old_hook_lsig_matches) {
/* We need to restore the old hook_lsig_matches */
cli_bitset_free(ctx->hook_lsig_matches); // safe to call, even if NULL
ctx->hook_lsig_matches = old_hook_lsig_matches;
}
Added additional ex_scan_callbacks test and fixed a couple related bugs Improvements to the ex_scan_callbacks.c program: - Print the verdict enum variant names to be more explicit. - Add the file_props callback (aka metadata JSON) with --gen-json option. - Add a --debug option. - Use '-' in option names instead of '_' to be consistent with other programs. - Add option to disable allmatch, which I named --one-match. :) Tests: Add ex_scan_callbacks test where --allmatch is disabled. Verify that CL_VIRUS is returned when a match occurs. I found a few bugs and inconsistencies from this test and went and fixed them, and improved the clamav.h function comments as well. Largely this resulted in cleanup in `cli_magic_scan()` to make sure we don't accidentally overwrite the return code. But it also meant making sure that callback functions which are supposed to trust a file actually clear the evidence/verdict and don't return CL_VIRUS.
2025-07-29 00:30:47 -04:00
return status;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
cl_error_t cli_magic_scan_desc_type(int desc, const char *filepath, cli_ctx *ctx, cli_file_t type,
const char *name, uint32_t attributes)
libclamav: allow logical sigs to be used as file type sigs (bb#2228)
2010-08-24 12:28:16 +02:00
{
BB#3737 - Value too large for specified data type Create compile-time preprocessor defines for switching from calling stat() to stat64(). Add --enable-stat64 switch in configure script.
2012-07-16 15:36:49 -04:00
STATBUF sb;
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fmap_t *new_map = NULL;
bb12262 - Fix to address potential use-after-free bug in scanner code relating to the filenames for nested files.
2019-01-30 15:01:59 -05:00
if (!ctx) {
return CL_EARG;
}
fmapify matcher/magic_scan partially
2011-06-14 11:11:05 +03:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_dbgmsg("in cli_magic_scan_desc_type (recursion_level: %u/%u)\n", ctx->recursion_level, ctx->engine->max_recursion_level);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (FSTAT(desc, &sb) == -1) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_errmsg("cli_magic_scan_desc_type: Can't fstat descriptor %d\n", desc);
bb12262 - Fix to address potential use-after-free bug in scanner code relating to the filenames for nested files.
2019-01-30 15:01:59 -05:00
status = CL_ESTAT;
bb12238 - Removing support for deprecated readdir_r() function. The readdir() function is thread safe so long as you don't share a dir object between threads. If you do, it requires a mutex.
2019-07-09 09:21:34 -07:00
goto done;
fmapify matcher/magic_scan partially
2011-06-14 11:11:05 +03:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (sb.st_size <= 5) {
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
cli_dbgmsg("cli_magic_scan_desc_type: Small data (%u bytes)\n", (unsigned int)sb.st_size);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
bb12238 - Removing support for deprecated readdir_r() function. The readdir() function is thread safe so long as you don't share a dir object between threads. If you do, it requires a mutex.
2019-07-09 09:21:34 -07:00
goto done;
fmapify matcher/magic_scan partially
2011-06-14 11:11:05 +03:00
}
perf_start(ctx, PERFT_MAP);
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
new_map = fmap_new(desc, 0, sb.st_size, name, filepath);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
perf_stop(ctx, PERFT_MAP);
if (NULL == new_map) {
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
cli_errmsg("cli_magic_scan_desc_type: CRITICAL: fmap_new() failed\n");
bb12262 - Fix to address potential use-after-free bug in scanner code relating to the filenames for nested files.
2019-01-30 15:01:59 -05:00
status = CL_EMEM;
bb12238 - Removing support for deprecated readdir_r() function. The readdir() function is thread safe so long as you don't share a dir object between threads. If you do, it requires a mutex.
2019-07-09 09:21:34 -07:00
goto done;
fmapify matcher/magic_scan partially
2011-06-14 11:11:05 +03:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_recursion_stack_push(ctx, new_map, type, true, attributes); /* Perform scan with child fmap */
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (CL_SUCCESS != status) {
cli_dbgmsg("Failed to scan fmap.\n");
goto done;
}
fmapify matcher/magic_scan partially
2011-06-14 11:11:05 +03:00
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
status = cli_magic_scan(ctx, type);
fmapify matcher/magic_scan partially
2011-06-14 11:11:05 +03:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
(void)cli_recursion_stack_pop(ctx); /* Restore the parent fmap */
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
bb12262 - Fix to address potential use-after-free bug in scanner code relating to the filenames for nested files.
2019-01-30 15:01:59 -05:00
done:
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (NULL != new_map) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(new_map);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
}
bb12262 - Fix to address potential use-after-free bug in scanner code relating to the filenames for nested files.
2019-01-30 15:01:59 -05:00
return status;
libclamav: allow logical sigs to be used as file type sigs (bb#2228)
2010-08-24 12:28:16 +02:00
}
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
cl_error_t cli_magic_scan_desc(int desc, const char *filepath, cli_ctx *ctx, const char *name, uint32_t attributes)
bb #1570: partition typing and HFS+
2013-09-17 16:45:48 -04:00
{
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
return cli_magic_scan_desc_type(desc, filepath, ctx, CL_TYPE_ANY, name, attributes);
bb #1570: partition typing and HFS+
2013-09-17 16:45:48 -04:00
}
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
/**
* @brief Scan an offset/length into a file map.
*
* Magic-scan some portion of an existing fmap.
*
* @param map File map.
* @param offset Offset into file map.
* @param length Length from offset.
* @param ctx Scanning context structure.
* @param type CL_TYPE of data to be scanned.
* @param name (optional) Original name of the file (to set fmap name metadata)
* @return int CL_SUCCESS, or an error code.
*/
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
static cl_error_t magic_scan_nested_fmap_type(cl_fmap_t *map, size_t offset, size_t length, cli_ctx *ctx,
cli_file_t type, const char *name, uint32_t attributes)
clamd: add ExtendedDetectionInfo (bb#1228, #1626)
2010-07-06 19:46:55 +02:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t status = CL_SUCCESS;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
fmap_t *new_map = NULL;
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
cli_dbgmsg("magic_scan_nested_fmap_type: [0, +%zu), [%zu, +%zu)\n",
map->len, offset, length);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Fix fmap-duplicate performance issue The fmap_duplicate function is used create a new fmap with a view into an existing fmap. When the new view is a different size than the old fmap, a new hash must be calculated for the duplicate fmap. However, when the duplicated fmap is the same size as the original fmap, the hash will be the same and there's no point recalculating. The issue is apparent when scanning large EXE files because the hash was being calculated at the beginning and end of the scan. Digging into this issue revealed that hash calculations for fmaps were also being performed at the wrong place. For scans of maps we use fmap_duplicate() early in the process to apply the name API argument to the duplicate fmap. Fixing the logic so we doing recalculate the hash revealed that we never calculated hashes for fmap's created from buffers in the first place, so that also had to be fixed be relocating where the hash is calculated. I also found that fmap_duplicate()'s offset argument used an off_t, though it and all caller offsets are not allowed to be negative. This was a bit of tangent to fix a bunch of off_t variables and paramters that should've been size_t. Added a couple unit tests to verify that making duplicate fmaps, and duplicate-duplicate fmaps works as expected after the change. Changed CLI_ISCONTAINED() and CLI_ISCONTAINED2() macros to cast to size_t, because pointers and buffer sizes may not be negative, and these two macros do not rely on substraction.
2021-01-23 16:41:41 -08:00
if (offset >= map->len) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_dbgmsg("magic_scan_nested_fmap_type: Invalid offset: %zu\n", offset);
goto done;
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
}
if (!length)
length = map->len - offset;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
if (length > map->len - offset) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_dbgmsg("magic_scan_nested_fmap_type: Data truncated: %zu -> %zu\n",
Fix fmap-duplicate performance issue The fmap_duplicate function is used create a new fmap with a view into an existing fmap. When the new view is a different size than the old fmap, a new hash must be calculated for the duplicate fmap. However, when the duplicated fmap is the same size as the original fmap, the hash will be the same and there's no point recalculating. The issue is apparent when scanning large EXE files because the hash was being calculated at the beginning and end of the scan. Digging into this issue revealed that hash calculations for fmaps were also being performed at the wrong place. For scans of maps we use fmap_duplicate() early in the process to apply the name API argument to the duplicate fmap. Fixing the logic so we doing recalculate the hash revealed that we never calculated hashes for fmap's created from buffers in the first place, so that also had to be fixed be relocating where the hash is calculated. I also found that fmap_duplicate()'s offset argument used an off_t, though it and all caller offsets are not allowed to be negative. This was a bit of tangent to fix a bunch of off_t variables and paramters that should've been size_t. Added a couple unit tests to verify that making duplicate fmaps, and duplicate-duplicate fmaps works as expected after the change. Changed CLI_ISCONTAINED() and CLI_ISCONTAINED2() macros to cast to size_t, because pointers and buffer sizes may not be negative, and these two macros do not rely on substraction.
2021-01-23 16:41:41 -08:00
length, map->len - offset);
length = map->len - offset;
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
}
if (length <= 5) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_dbgmsg("magic_scan_nested_fmap_type: Small data (%zu bytes)\n", length);
goto done;
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
new_map = fmap_duplicate(map, offset, length, name);
if (NULL == new_map) {
cli_dbgmsg("magic_scan_nested_fmap_type: Failed to duplicate fmap for scan of fmap subsection\n");
goto done;
}
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
status = cli_recursion_stack_push(ctx, new_map, type, false, attributes); /* Perform scan with child fmap */
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (CL_SUCCESS != status) {
cli_dbgmsg("magic_scan_nested_fmap_type: Failed to add map to recursion stack for magic scan.\n");
goto done;
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = cli_magic_scan(ctx, type);
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
(void)cli_recursion_stack_pop(ctx); /* Restore the parent fmap */
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
done:
if (NULL != new_map) {
free_duplicate_fmap(new_map); /* This fmap is just a duplicate. */
}
return status;
clamd: add ExtendedDetectionInfo (bb#1228, #1626)
2010-07-06 19:46:55 +02:00
}
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
/* For map scans that may be forced to disk */
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
cl_error_t cli_magic_scan_nested_fmap_type(cl_fmap_t *map, size_t offset, size_t length, cli_ctx *ctx,
cli_file_t type, const char *name, uint32_t attributes)
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
{
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t ret = CL_SUCCESS;
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
Fix fmap-duplicate performance issue The fmap_duplicate function is used create a new fmap with a view into an existing fmap. When the new view is a different size than the old fmap, a new hash must be calculated for the duplicate fmap. However, when the duplicated fmap is the same size as the original fmap, the hash will be the same and there's no point recalculating. The issue is apparent when scanning large EXE files because the hash was being calculated at the beginning and end of the scan. Digging into this issue revealed that hash calculations for fmaps were also being performed at the wrong place. For scans of maps we use fmap_duplicate() early in the process to apply the name API argument to the duplicate fmap. Fixing the logic so we doing recalculate the hash revealed that we never calculated hashes for fmap's created from buffers in the first place, so that also had to be fixed be relocating where the hash is calculated. I also found that fmap_duplicate()'s offset argument used an off_t, though it and all caller offsets are not allowed to be negative. This was a bit of tangent to fix a bunch of off_t variables and paramters that should've been size_t. Added a couple unit tests to verify that making duplicate fmaps, and duplicate-duplicate fmaps works as expected after the change. Changed CLI_ISCONTAINED() and CLI_ISCONTAINED2() macros to cast to size_t, because pointers and buffer sizes may not be negative, and these two macros do not rely on substraction.
2021-01-23 16:41:41 -08:00
cli_dbgmsg("cli_magic_scan_nested_fmap_type: [%zu, +%zu)\n", offset, length);
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
if (offset >= map->len) {
Fix fmap-duplicate performance issue The fmap_duplicate function is used create a new fmap with a view into an existing fmap. When the new view is a different size than the old fmap, a new hash must be calculated for the duplicate fmap. However, when the duplicated fmap is the same size as the original fmap, the hash will be the same and there's no point recalculating. The issue is apparent when scanning large EXE files because the hash was being calculated at the beginning and end of the scan. Digging into this issue revealed that hash calculations for fmaps were also being performed at the wrong place. For scans of maps we use fmap_duplicate() early in the process to apply the name API argument to the duplicate fmap. Fixing the logic so we doing recalculate the hash revealed that we never calculated hashes for fmap's created from buffers in the first place, so that also had to be fixed be relocating where the hash is calculated. I also found that fmap_duplicate()'s offset argument used an off_t, though it and all caller offsets are not allowed to be negative. This was a bit of tangent to fix a bunch of off_t variables and paramters that should've been size_t. Added a couple unit tests to verify that making duplicate fmaps, and duplicate-duplicate fmaps works as expected after the change. Changed CLI_ISCONTAINED() and CLI_ISCONTAINED2() macros to cast to size_t, because pointers and buffer sizes may not be negative, and these two macros do not rely on substraction.
2021-01-23 16:41:41 -08:00
cli_dbgmsg("Invalid offset: %zu\n", offset);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
return CL_SUCCESS;
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ctx->engine->engine_options & ENGINE_OPTIONS_FORCE_TO_DISK) {
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
/*
* Force to disk!
*
* Write the offset + length section of the fmap to disk, and scan it.
*/
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
const uint8_t *mapdata = NULL;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
char *tempfile = NULL;
int fd = -1;
size_t nread = 0;
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
libclamav: Have cli_map_scan check for truncation for dump-to-file case
2013-12-11 15:30:40 -05:00
/* Then check length */
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
if (!length) {
/* Caller didn't specify len, use rest of the map */
length = map->len - offset;
}
if (length > map->len - offset) {
cli_dbgmsg("cli_magic_scan_nested_fmap_type: Data truncated: %zu -> %zu\n", length, map->len - offset);
length = map->len - offset;
libclamav: Have cli_map_scan check for truncation for dump-to-file case
2013-12-11 15:30:40 -05:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (length <= 5) {
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cli_dbgmsg("cli_magic_scan_nested_fmap_type: Small data (%u bytes)\n", (unsigned int)length);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
return CL_SUCCESS;
libclamav: Have cli_map_scan check for truncation for dump-to-file case
2013-12-11 15:30:40 -05:00
}
Fix fmap handle_gets() page arithmetic The fixes to the fmap bounds for nested (duplicate) fmaps added recently introduced a subtle arithmetic bug that was detected by OSS-Fuzz: ```c scanat = m->nested_offset + *at % m->pgsz; ``` should have been: ```c scanat = (m->nested_offset + *at) % m->pgsz; ``` Without the parenthesis, `scanat` could be > `m->pgsz`, which would overflow in the subsequent `memchr()` call. See: - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40452 - https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40455 This commit also tightens up some of the other bounds checks done with `CLI_ISCONTAINED()` macro so the check limits the bounds to the nested fmap and not the original map. In addition, I've added a `CLI_ISCONTAINED_0_TO()` macro that removes checks when the "bigger" buffer starts at offset 0. This should silence a bunch of (benign) warnings and medium severity Coverity issues. There is also a possible use of an uninitialized variable (`old_hook_lsig_matches`) in `cli_magic_scan()`. Finally, I also removed an unecessary NULL-check on `filebase` in `fmap_dup_to_file()` that Coverity was unhappy with.
2021-10-28 16:58:21 -07:00
if (!CLI_ISCONTAINED_0_TO(map->len, offset, length)) {
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
cli_dbgmsg("cli_magic_scan_nested_fmap_type: map error occurred [%zu, %zu] not within [0, %zu]\n", offset, length, map->len);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
return CL_SUCCESS;
libclamav: Have cli_map_scan check for truncation for dump-to-file case
2013-12-11 15:30:40 -05:00
}
/* Length checked, now get map */
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
mapdata = fmap_need_off_once_len(map, offset, length, &nread);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!mapdata || (nread != length)) {
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cli_errmsg("cli_magic_scan_nested_fmap_type: could not map sub-file\n");
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
return CL_EMAP;
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
ret = cli_gentempfd(ctx->this_layer_tmpdir, &tempfile, &fd);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ret != CL_SUCCESS) {
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
return ret;
}
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cli_dbgmsg("cli_magic_scan_nested_fmap_type: writing nested map content to temp file %s\n", tempfile);
Argument and return types for fmap_readn(), cli_writen(), cli_readn() converted to use size_t instead of int.
2019-05-04 15:54:54 -04:00
if (cli_writen(fd, mapdata, length) == (size_t)-1) {
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cli_errmsg("cli_magic_scan_nested_fmap_type: cli_writen error writing subdoc temporary file.\n");
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
ret = CL_EWRITE;
}
/* scan the temp file */
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_desc_type(fd, tempfile, ctx, type, name, attributes);
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
/* remove the temp file, if needed */
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (fd >= 0) {
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
close(fd);
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!ctx->engine->keeptmp) {
if (cli_unlink(tempfile)) {
Rename core scanning functions Many of the core scanning functions' names no longer represent their specific purpose or arguments. This commit aims to make the names more intuitive. Names are now prefixed with "magic" if they involve file-typing and file-type parsing. In addition, each function now includes the type of input being scanned whether its "desc", "fmap", or "buff". Some of the APIs also now specify "type" to indicate that a type other than "ANY" may be passed in to select the type rather than use file type magic for type recognition. | current name | new name | | ------------------------- | --------------------------------- | | magic_scandesc() | cli_magic_scan() | | cli_magic_scandesc_type() | <delete> | | cli_magic_scandesc() | cli_magic_scan_desc() | | cli_base_scandesc() | cli_magic_scan_desc_type() | | cli_partition_scandesc() | <delete> | | cli_map_scandesc() | magic_scan_nested_fmap_type() | | cli_map_scan() | cli_magic_scan_nested_fmap_type() | | cli_mem_scandesc() | cli_magic_scan_buff() | | cli_scanbuff() | cli_scan_buff() | | cli_scandesc() | cli_scan_desc() | | cli_fmap_scandesc() | cli_scan_fmap() | | cli_scanfile() | cli_magic_scan_file() | | cli_scandir() | cli_magic_scan_dir() | | cli_filetype2() | cli_determine_fmap_type() | | cli_filetype() | cli_compare_ftm_file() | | cli_partitiontype() | cli_compare_ftm_partition() | | cli_scanraw() | scanraw() |
2020-03-21 14:15:28 -04:00
cli_errmsg("cli_magic_scan_nested_fmap_type: error unlinking tempfile %s\n", tempfile);
Add ForceToDisk option for clamd and force-to-disk arg for clamscan
2013-11-08 17:10:43 -05:00
ret = CL_EUNLINK;
}
}
free(tempfile);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
/*
* Not forced to disk.
*
* Just use nested map by scanning given fmap at offset + length.
*/
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = magic_scan_nested_fmap_type(map, offset, length, ctx, type, name, attributes);
Introduce cli_map_scandesc to scan a portion of the existing file And switch CPIO, MACHO, and SWF to use it. Now they no longer need to dump a tempfile and remap. To investigate if it is possible to do this with TAR.
2011-06-17 23:08:31 +03:00
}
fmap scan API
2011-06-14 21:49:39 +03:00
return ret;
}
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
cl_error_t cli_magic_scan_buff(const void *buffer, size_t length, cli_ctx *ctx, const char *name, uint32_t attributes)
cli_map_scandesc convenience API
2011-06-17 23:14:36 +03:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cl_error_t ret;
fmap_t *map = NULL;
map = fmap_open_memory(buffer, length, name);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (!map) {
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
return CL_EMAP;
cli_map_scandesc convenience API
2011-06-17 23:14:36 +03:00
}
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_nested_fmap_type(map, 0, length, ctx, CL_TYPE_ANY, name, attributes);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(map);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
cli_map_scandesc convenience API
2011-06-17 23:14:36 +03:00
return ret;
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
/**
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
* @brief The main function to initiate a scan of an fmap.
fuzz - 12166 - Fix for 4-byte out of bounds write wherein the an invalid struct pointer member variable is set to zero. The fix adds bounds checking to the Uniq storage 'add' function as well as error code checks. Included a lot of new inline documentation.
2019-01-22 14:05:05 -05:00
*
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
* @param map File map.
* @param filepath (optional, recommended) filepath of the open file descriptor or file map.
* @param[out] verdict_out A pointer to a cl_verdict_t that will be set to the scan verdict.
* You should check the verdict even if the function returns an error.
* @param[out] last_alert_out Will be set to a statically allocated (i.e. needs not be freed) signature name if the scan matches against a signature.
* @param[out] scanned_out (Optional) The number of bytes scanned.
* @param engine The scanning engine.
* @param scanoptions Scanning options.
* @param[in,out] context (Optional) An application-defined context struct, opaque to libclamav.
* May be used within your callback functions.
* @param hash_hint (Optional) A NULL terminated string of the file hash so that
* libclamav does not need to calculate it.
* @param[out] hash_out (Optional) A NULL terminated string of the file hash.
* The caller is responsible for freeing this string.
* @param hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`.
* Supported algorithms are "md5", "sha1", "sha2-256".
* Required only if you provide a `hash_hint` or want to receive a `hash_out`.
* @param file_type_hint (Optional) A NULL terminated string of the file type hint.
* E.g. "pe", "elf", "zip", etc.
* You may also use ClamAV type names such as "CL_TYPE_PE".
* ClamAV will ignore the hint if it is not familiar with the specified type.
* @param file_type_out (Optional) A NULL terminated string of the file type
* of the top layer as determined by ClamAV.
* Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE".
* The caller is responsible for freeing this string.
* @return cl_error_t CL_SUCCESS if no error occured.
* Otherwise a CL_E* error code.
* Does NOT return CL_VIRUS for a signature match. Check the `verdict_out` parameter instead.
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
*/
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
static cl_error_t scan_common(
cl_fmap_t *map,
const char *filepath,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_verdict_t *verdict_out,
const char **last_alert_out,
uint64_t *scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context,
const char *hash_hint,
char **hash_out,
const char *hash_alg,
const char *file_type_hint,
char **file_type_out)
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
{
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cl_error_t status = CL_SUCCESS;
cl_error_t ret;
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_ctx ctx = {0};
bb #10970 - Force a filesize limit of UINT_MAX - 2
2014-04-11 13:41:18 -04:00
Fix typos (no functional changes)
2024-01-19 09:08:36 -08:00
bool logg_initialized = false;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
char *target_basename = NULL;
char *new_temp_prefix = NULL;
size_t new_temp_prefix_len;
char *new_temp_path = NULL;
time_t current_time;
struct tm tm_struct;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
size_t num_potentially_unwanted_indicators = 0;
// The default type is SHA2-256.
cli_hash_type_t requested_hash_type = CLI_HASH_SHA2_256;
// The type of the file being scanned.
cli_file_t file_type = CL_TYPE_ANY;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (NULL == map || NULL == scanoptions || NULL == verdict_out || NULL == last_alert_out || NULL == engine) {
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
return CL_ENULLARG;
bb #10970 - Force a filesize limit of UINT_MAX - 2
2014-04-11 13:41:18 -04:00
}
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
/* Initialize output variables */
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
*verdict_out = CL_VERDICT_NOTHING_FOUND;
*last_alert_out = NULL;
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
// If the caller provided a file type hint, we make a best effort to use it.
if (file_type_hint) {
file_type = cli_ftcode_human_friendly(file_type_hint);
if (CL_TYPE_ERROR == file_type) {
cli_dbgmsg("scan_common: Unsupported file type hint: %s. Will treat it as unknown (CL_TYPE_ANY)\n", file_type_hint);
file_type = CL_TYPE_ANY;
}
}
if (NULL != hash_out) {
*hash_out = NULL;
}
if (NULL != hash_alg) {
// Set the fmap hash for the given algorithm.
libclamav: Fix issue reporting trusted verdicts If the outermost layer is trusted (e.g. using an FP signature), the verdict passed back by the `cl_scan*_ex()` functions should be CL_VERDICT_TRUSTED. To ensure this, and other correct verdicts, I moved the logic setting the verdict to occur when adding indicators, or trusting a given layer. Then at the end of a scan, it will set the output verdict parameter to the top level verdict. This commit also: * Fixes a bug in the `ex_scan_callbacks` program where a crash would happen when a hash was retrieved for an inner layer, but isn't for the container. * Added debug logs whenever a hash is calculated or set, printing the hash type and hash string. * When a layer is trusted, in addition to removing evidence for that layer, it will also index the metadata JSON (if that feature is enabled) and will rename any "Viruses" to "IgnoredAlerts", and rename "ContainedIndicators" to "IgnoredContainedIndicators". * Fixed an issue where setting the hash algorithm with extra characters, like setting to "sha256789" would ignore the extra characters, and report the hash type as the same. It will now fail if the string length differs from the known hash algorithm.
2025-08-07 13:22:57 -04:00
if (3 == strlen(hash_alg) && (0 == strncmp(hash_alg, "md5", 3) || (0 == strncmp(hash_alg, "MD5", 3)))) {
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
requested_hash_type = CLI_HASH_MD5;
libclamav: Fix issue reporting trusted verdicts If the outermost layer is trusted (e.g. using an FP signature), the verdict passed back by the `cl_scan*_ex()` functions should be CL_VERDICT_TRUSTED. To ensure this, and other correct verdicts, I moved the logic setting the verdict to occur when adding indicators, or trusting a given layer. Then at the end of a scan, it will set the output verdict parameter to the top level verdict. This commit also: * Fixes a bug in the `ex_scan_callbacks` program where a crash would happen when a hash was retrieved for an inner layer, but isn't for the container. * Added debug logs whenever a hash is calculated or set, printing the hash type and hash string. * When a layer is trusted, in addition to removing evidence for that layer, it will also index the metadata JSON (if that feature is enabled) and will rename any "Viruses" to "IgnoredAlerts", and rename "ContainedIndicators" to "IgnoredContainedIndicators". * Fixed an issue where setting the hash algorithm with extra characters, like setting to "sha256789" would ignore the extra characters, and report the hash type as the same. It will now fail if the string length differs from the known hash algorithm.
2025-08-07 13:22:57 -04:00
} else if (4 == strlen(hash_alg) && (0 == strncmp(hash_alg, "sha1", 4) || (0 == strncmp(hash_alg, "SHA1", 4)))) {
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
requested_hash_type = CLI_HASH_SHA1;
libclamav: Fix issue reporting trusted verdicts If the outermost layer is trusted (e.g. using an FP signature), the verdict passed back by the `cl_scan*_ex()` functions should be CL_VERDICT_TRUSTED. To ensure this, and other correct verdicts, I moved the logic setting the verdict to occur when adding indicators, or trusting a given layer. Then at the end of a scan, it will set the output verdict parameter to the top level verdict. This commit also: * Fixes a bug in the `ex_scan_callbacks` program where a crash would happen when a hash was retrieved for an inner layer, but isn't for the container. * Added debug logs whenever a hash is calculated or set, printing the hash type and hash string. * When a layer is trusted, in addition to removing evidence for that layer, it will also index the metadata JSON (if that feature is enabled) and will rename any "Viruses" to "IgnoredAlerts", and rename "ContainedIndicators" to "IgnoredContainedIndicators". * Fixed an issue where setting the hash algorithm with extra characters, like setting to "sha256789" would ignore the extra characters, and report the hash type as the same. It will now fail if the string length differs from the known hash algorithm.
2025-08-07 13:22:57 -04:00
} else if ((8 == strlen(hash_alg) && (0 == strncmp(hash_alg, "sha2-256", 8) || (0 == strncmp(hash_alg, "SHA2-256", 8)))) ||
(6 == strlen(hash_alg) && (0 == strncmp(hash_alg, "sha256", 6) || (0 == strncmp(hash_alg, "SHA256", 6))))) {
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
requested_hash_type = CLI_HASH_SHA2_256;
} else {
cli_errmsg("scan_common: Unsupported hash algorithm: %s\n", hash_alg);
status = CL_EARG;
goto done;
}
}
// If hash_hint is provided, we need to check if the hash_alg is valid.
if (NULL != hash_hint) {
uint8_t hash[CLI_HASHLEN_MAX] = {0};
size_t hash_string_len = strlen(hash_hint);
if (hash_string_len != cli_hash_len(requested_hash_type) * 2) {
cli_errmsg("scan_common: hash_hint provided, but its length (%zu) does not match the expected length for %s (%zu).\n",
hash_string_len, hash_alg, cli_hash_len(requested_hash_type) * 2);
status = CL_EARG;
goto done;
}
// Convert the hash_hint string to a binary hash.
ret = cli_hexstr_to_bytes(hash_hint, hash_string_len, hash);
if (ret != CL_SUCCESS) {
cli_errmsg("scan_common: hash_hint provided, but it is not a valid hex string.\n");
status = CL_EARG;
goto done;
}
// Set the fmap hash for the given algorithm.
if (CL_SUCCESS != fmap_set_hash(map, hash, requested_hash_type)) {
cli_errmsg("scan_common: Failed to set fmap hash for %s.\n", hash_alg);
status = CL_EARG;
goto done;
}
cli_dbgmsg("scan_common: recorded %s hash hint: %s\n", cli_hash_name(requested_hash_type), hash_hint);
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ctx.engine = engine;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
ctx.scanned = scanned_out;
Refine max-allocation and safer-allocation function and macro names We add the _OR_GOTO_DONE suffix to the macros that go to done if the allocation fails. This makes it obvious what is different about the macro versus the equivalent function, and that error handling is built-in. Renamed the cli_strdup to safer_strdup to make it obvious that it exists because it is safer than regular strdup. Regular strdup doesn't have the NULL check before trying to dup, and so may result in a NULL-deref crash. Also remove unused STRDUP (_OR_GOTO_DONE) macro, since the one with the NULL-check is preferred.
2024-01-09 17:44:33 -05:00
CLI_MALLOC_OR_GOTO_DONE(ctx.options, sizeof(struct cl_scan_options), status = CL_EMEM);
Decrypt old Excel default-password files Add support for detecting and decrypting OLE2-based Excel XLS files that are encrypted with the default password 'VelvetSweatshop'.
2022-10-21 19:27:36 -04:00
Restructured scan options flags from a single bitflag field to a structure containing multiple bitflag fields. This also required adding a new function to the bytecode API to get scan options a la carte, and modifying the existing function to hand back scan options in the old/deprecated uint32_t bitflag format. Re-generated bytecode iface header files. Updated libclamav documentation detailing new scan options structure. Renamed references to 'algorithmic' detection to 'heuristic' detection. Renaming references to 'properties' to 'collect metadata'. Renamed references to 'scan all' to 'scan all match'. Renamed a couple of 'Hueristic.*' signature names as 'Heuristics.*' signatures (plural) to match majority of other heuristics.
2018-07-20 22:28:48 -04:00
memcpy(ctx.options, scanoptions, sizeof(struct cl_scan_options));
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ctx.dconf = (struct cli_dconf *)engine->dconf;
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
ctx.cb_ctx = context;
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (!(ctx.hook_lsig_matches = cli_bitset_init())) {
status = CL_EMEM;
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
goto done;
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
ctx.recursion_stack_size = ctx.engine->max_recursion_level;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
ctx.recursion_stack = calloc(sizeof(cli_scan_layer_t), ctx.recursion_stack_size);
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (!ctx.recursion_stack) {
status = CL_EMEM;
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
goto done;
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
}
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
// ctx was memset, so recursion_level starts at 0.
ctx.recursion_stack[ctx.recursion_level].fmap = map;
Fix issues reading from uncompressed nested files The fmap module provides a mechanism for creating a mapping into an existing map at an offset and length that's used when a file is found with an uncompressed archive or when embedded files are found with embedded file type recognition in scanraw(). This is the "fmap_duplicate()" function. Duplicate fmaps just reference the original fmap's 'data' or file handle/descriptor while allowing the caller to treat it like a new map using offsets and lengths that don't account for the original/actual file dimensions. fmap's keep track of this with m->nested_offset & m->real_len, which admittedly have confusing names. I found incorrect uses of these in a handful of locations. Notably: - In cli_magic_scan_nested_fmap_type(). The force-to-disk feature would have been checking incorrect sizes and may have written incorrect offsets for duplicate fmaps. - In XDP parser. - A bunch of places from the previous commit when making dupe maps. This commit fixes those and adds lots of documentation to the fmap.h API to try to prevent confusion in the future. nested_offset should never be referenced outside of fmap.c/h. The fmap_* functions for accessing or reading map data have two implementations, mem_* or handle_*, depending the data source. I found issues with some of these so I made a unit test that covers each of the functions I'm concerned about for both types of data sources and for both original fmaps and nested/duplicate fmaps. With the tests, I found and fixed issues in these fmap functions: - handle_need_offstr(): must account for the nested_offset in dupe maps. - handle_gets(): must account for nested_offset and use len & real_len correctly. - mem_need_offstr(): must account for nested_offset in dupe maps. - mem_gets(): must account for nested_offset and use len & real_len correctly. Moved CDBRANGE() macro out of function definition so for better legibility. Fixed a few warnings.
2021-10-03 14:13:55 -07:00
ctx.recursion_stack[ctx.recursion_level].size = map->len;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
ctx.recursion_stack[ctx.recursion_level].type = CL_TYPE_ANY;
ctx.fmap = ctx.recursion_stack[ctx.recursion_level].fmap;
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
Add some performance measurement code. Default off, activated by --dev-performance, and always on in c4w.
2011-02-14 19:19:20 +02:00
perf_init(&ctx);
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
if (ctx.engine->maxscantime != 0) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (gettimeofday(&ctx.time_limit, NULL) == 0) {
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
uint32_t secs = ctx.engine->maxscantime / 1000;
uint32_t usecs = (ctx.engine->maxscantime % 1000) * 1000;
Add timeout facility for file property scanning and clamscan timeout testing parameter.
2014-06-13 16:11:15 -04:00
ctx.time_limit.tv_sec += secs;
ctx.time_limit.tv_usec += usecs;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (ctx.time_limit.tv_usec >= 1000000) {
Add timeout facility for file property scanning and clamscan timeout testing parameter.
2014-06-13 16:11:15 -04:00
ctx.time_limit.tv_usec -= 1000000;
ctx.time_limit.tv_sec++;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
Add timeout facility for file property scanning and clamscan timeout testing parameter.
2014-06-13 16:11:15 -04:00
char buf[64];
Adds --max-scantime clamscan option and MaxScanTime clamd config option. --max-scantime replaces the --timelimit clamscan option that had been experimental. Default max-scantime set to 2 minutes (120000 milliseconds).
2019-08-16 17:18:59 -07:00
cli_dbgmsg("scan_common: gettimeofday error: %s\n", cli_strerror(errno, buf, 64));
Add timeout facility for file property scanning and clamscan timeout testing parameter.
2014-06-13 16:11:15 -04:00
}
}
Fix file access issue if in low privelege process Removing problematic call to convert file descriptors to filepaths. Added filename and tempfile names to scandesc calls in clamd. Added a general scan option to treat the scan engine as unprivileged, meaning that the scan engine will not have read access to the file. Added check to drop a temp file for RAR's where the we don't have read access to the filepath provided (i.e. unprivileged is set, or access() check fails).
2020-01-23 17:42:33 -08:00
if (filepath != NULL) {
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
ctx.target_filepath = strdup(filepath);
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
}
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
/*
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
* Create a tmp sub-directory for the temp files generated by this scan.
*
* If keeptmp (LeaveTemporaryFiles / --leave-temps) is enabled, we'll include the
* basename in the tmp directory.
* If keeptmp is not enabled, we'll just call it "scantemp".
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
*/
current_time = time(NULL);
#ifdef _WIN32
if (0 != localtime_s(&tm_struct, &current_time)) {
#else
if (!localtime_r(&current_time, &tm_struct)) {
#endif
cli_errmsg("scan_common: Failed to get local time.\n");
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_ESTAT;
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
goto done;
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ctx.engine->engine_options & ENGINE_OPTIONS_TMPDIR_RECURSION)) {
if ((ctx.engine->keeptmp) &&
(NULL != ctx.target_filepath) &&
(CL_SUCCESS == cli_basename(ctx.target_filepath, strlen(ctx.target_filepath), &target_basename, true /* posix_support_backslash_pathsep */))) {
/* Include the basename in the temp directory */
new_temp_prefix_len = strlen("YYYYMMDD_HHMMSS-") + strlen(target_basename);
new_temp_prefix = cli_max_calloc(1, new_temp_prefix_len + 1);
if (!new_temp_prefix) {
cli_errmsg("scan_common: Failed to allocate memory for temp directory name.\n");
status = CL_EMEM;
goto done;
}
strftime(new_temp_prefix, new_temp_prefix_len + 1, "%Y%m%d_%H%M%S-", &tm_struct);
strcpy(new_temp_prefix + strlen("YYYYMMDD_HHMMSS-"), target_basename);
} else {
/* Just use date */
new_temp_prefix_len = strlen("YYYYMMDD_HHMMSS-scantemp");
new_temp_prefix = cli_max_calloc(1, new_temp_prefix_len + 1);
if (!new_temp_prefix) {
cli_errmsg("scan_common: Failed to allocate memory for temp directory name.\n");
status = CL_EMEM;
goto done;
}
strftime(new_temp_prefix, new_temp_prefix_len + 1, "%Y%m%d_%H%M%S-scantemp", &tm_struct);
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
/* Place the new temp sub-directory within the configured temp directory */
new_temp_path = cli_gentemp_with_prefix(ctx.engine->tmpdir, new_temp_prefix);
free(new_temp_prefix);
if (NULL == new_temp_path) {
cli_errmsg("scan_common: Failed to generate temp directory name.\n");
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EMEM;
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
goto done;
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
ctx.recursion_stack[ctx.recursion_level].tmpdir = new_temp_path;
ctx.this_layer_tmpdir = new_temp_path;
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if (mkdir(ctx.this_layer_tmpdir, 0700)) {
cli_errmsg("Can't create temporary directory for scan: %s.\n", ctx.this_layer_tmpdir);
status = CL_EACCES;
goto done;
}
} else {
/*
* Use the configured temp directory.
* Making a unique subdirectory per scan is slower, and particularly slow on Windows.
*/
ctx.recursion_stack[ctx.recursion_level].tmpdir = ctx.engine->tmpdir;
ctx.this_layer_tmpdir = ctx.engine->tmpdir;
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
}
Default off, you can turn on via 'DevLiblog'. This also replaces the cli_*_stats variants with a callback for stats, so that clamd can call the cl_*_callback variants instead, and pass the filename as context.
2010-11-02 12:26:33 +02:00
cli_logg_setup(&ctx);
Fix typos (no functional changes)
2024-01-19 09:08:36 -08:00
logg_initialized = true;
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
// Assign a unique object_id to the new container.
ctx.recursion_stack[ctx.recursion_level].object_id = ctx.object_count;
ctx.object_count++;
if (ctx.options->general & CL_SCAN_GENERAL_COLLECT_METADATA) {
ctx.metadata_json = json_object_new_object();
if (NULL == ctx.metadata_json) {
cli_errmsg("scan_common: no memory for json properties object\n");
status = CL_EMEM;
goto done;
}
/* Set the convenience pointer to the current properties object */
ctx.recursion_stack[ctx.recursion_level].metadata_json = ctx.metadata_json;
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
ctx.this_layer_metadata_json = ctx.metadata_json;
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
status = cli_jsonstr(ctx.metadata_json, "Magic", "CLAMJSONv0");
if (status != CL_SUCCESS) {
cli_errmsg("scan_common: error setting Magic property in metadata.json\n");
goto done;
}
if (ctx.fmap->name) {
status = cli_jsonstr(ctx.metadata_json, "FileName", ctx.fmap->name);
if (status != CL_SUCCESS) {
cli_errmsg("scan_common: error setting FileName property in metadata.json\n");
goto done;
}
}
if (ctx.fmap->path) {
status = cli_jsonstr(ctx.metadata_json, "FilePath", ctx.fmap->path);
if (status != CL_SUCCESS) {
cli_errmsg("scan_common: error setting FilePath property in metadata.json\n");
goto done;
}
}
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
status = cli_jsonuint64(ctx.metadata_json, "FileSize", (uint64_t)ctx.fmap->len);
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (status != CL_SUCCESS) {
cli_errmsg("scan_common: error setting FileSize property in metadata.json\n");
goto done;
}
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
status = cli_jsonuint64(ctx.metadata_json, "ObjectID", (uint64_t)ctx.recursion_stack[ctx.recursion_level].object_id);
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (status != CL_SUCCESS) {
cli_errmsg("scan_common: error setting ObjectID property in metadata.json\n");
goto done;
}
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
/*
* DO THE SCAN!
*/
status = cli_magic_scan(&ctx, file_type);
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (ctx.options->general & CL_SCAN_GENERAL_COLLECT_METADATA && (ctx.metadata_json != NULL)) {
ole2: cleaned-up commented code json: added unique identifier for root filetype (RootFileType)
2014-06-30 11:44:02 -04:00
json_object *jobj;
const char *jstring;
/* set value of unique root object tag */
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (json_object_object_get_ex(ctx.metadata_json, "FileType", &jobj)) {
ole2: cleaned-up commented code json: added unique identifier for root filetype (RootFileType)
2014-06-30 11:44:02 -04:00
enum json_type type;
const char *jstr;
type = json_object_get_type(jobj);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (type == json_type_string) {
ole2: cleaned-up commented code json: added unique identifier for root filetype (RootFileType)
2014-06-30 11:44:02 -04:00
jstr = json_object_get_string(jobj);
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
cli_jsonstr(ctx.metadata_json, "RootFileType", jstr);
ole2: cleaned-up commented code json: added unique identifier for root filetype (RootFileType)
2014-06-30 11:44:02 -04:00
}
}
Add callback for user processing of json string and result of json scan.
2014-06-03 13:31:50 -04:00
/* serialize json properties to string */
Add EmbeddedObjects to JSON The metadata projecties JSON structure isn't recording file types found embedded within a file such as self-extracting (SFX) types and office document types (DOCX, PPTX, etc). This presents a problem... At present there's no way to know if the current file has ended and a few file is found tacked on to the end of the first file. If there were, we could simply check if the type found by the raw-scan exists within the first file, or after. If within the first, and the type is an archive then it's reasonable to conclude we're either observing zip headers (for SFXZIP detections) or other files that are not compressed. If the type ISN'T found within the first file, then we definitely have whole new file to parse and we should do so with cli_magic_scan() rather than only using these embedded type scanners. At present we can't ignore SFXZIP detections even if the original file type is a ZIP because we may have found two ZIPs appended together to evade detection (a legitimate trick). As a consequence, we will effectively parse every zip entry twice. The same issue applies to types found within non-compressed archives. This commit adds an EmbeddedObjects list to the metadata JSON object so that the existance of these types is noted. Additionally, this commit removes the two-part int64 cli_jsonint64() implementation as json_object_new_int64() should be available everywhere and the macro to detect such support was never set.
2020-03-21 11:36:53 -04:00
#ifdef JSON_C_TO_STRING_NOSLASHESCAPE
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
jstring = json_object_to_json_string_ext(ctx.metadata_json, JSON_C_TO_STRING_PRETTY | JSON_C_TO_STRING_NOSLASHESCAPE);
Add EmbeddedObjects to JSON The metadata projecties JSON structure isn't recording file types found embedded within a file such as self-extracting (SFX) types and office document types (DOCX, PPTX, etc). This presents a problem... At present there's no way to know if the current file has ended and a few file is found tacked on to the end of the first file. If there were, we could simply check if the type found by the raw-scan exists within the first file, or after. If within the first, and the type is an archive then it's reasonable to conclude we're either observing zip headers (for SFXZIP detections) or other files that are not compressed. If the type ISN'T found within the first file, then we definitely have whole new file to parse and we should do so with cli_magic_scan() rather than only using these embedded type scanners. At present we can't ignore SFXZIP detections even if the original file type is a ZIP because we may have found two ZIPs appended together to evade detection (a legitimate trick). As a consequence, we will effectively parse every zip entry twice. The same issue applies to types found within non-compressed archives. This commit adds an EmbeddedObjects list to the metadata JSON object so that the existance of these types is noted. Additionally, this commit removes the two-part int64 cli_jsonint64() implementation as json_object_new_int64() should be available everywhere and the macro to detect such support was never set.
2020-03-21 11:36:53 -04:00
#else
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
jstring = json_object_to_json_string_ext(ctx.metadata_json, JSON_C_TO_STRING_PRETTY);
Add EmbeddedObjects to JSON The metadata projecties JSON structure isn't recording file types found embedded within a file such as self-extracting (SFX) types and office document types (DOCX, PPTX, etc). This presents a problem... At present there's no way to know if the current file has ended and a few file is found tacked on to the end of the first file. If there were, we could simply check if the type found by the raw-scan exists within the first file, or after. If within the first, and the type is an archive then it's reasonable to conclude we're either observing zip headers (for SFXZIP detections) or other files that are not compressed. If the type ISN'T found within the first file, then we definitely have whole new file to parse and we should do so with cli_magic_scan() rather than only using these embedded type scanners. At present we can't ignore SFXZIP detections even if the original file type is a ZIP because we may have found two ZIPs appended together to evade detection (a legitimate trick). As a consequence, we will effectively parse every zip entry twice. The same issue applies to types found within non-compressed archives. This commit adds an EmbeddedObjects list to the metadata JSON object so that the existance of these types is noted. Additionally, this commit removes the two-part int64 cli_jsonint64() implementation as json_object_new_int64() should be available everywhere and the macro to detect such support was never set.
2020-03-21 11:36:53 -04:00
#endif
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (NULL == jstring) {
Add build support for libjson and initial json work for properties file.
2014-04-16 16:40:56 -04:00
cli_errmsg("scan_common: no memory for json serialization.\n");
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
status = CL_EMEM;
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
goto done;
}
cli_dbgmsg("%s\n", jstring);
if (status != CL_VIRUS) {
/*
* Run bytecode preclass hook.
*/
preclass: scanning condition includes ac_patterns and pcre_metas
2016-05-17 16:44:21 -04:00
struct cli_matcher *iroot = ctx.engine->root[13];
fixed an incorrect return value for magic_scandesc
2015-01-15 15:15:01 -08:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
struct cli_bc_ctx *bc_ctx = cli_bytecode_context_alloc();
if (!bc_ctx) {
cli_errmsg("scan_common: can't allocate memory for bc_ctx\n");
status = CL_EMEM;
} else {
cli_bytecode_context_setctx(bc_ctx, &ctx);
status = cli_bytecode_runhook(&ctx, ctx.engine, bc_ctx, BC_PRECLASS, map);
cli_bytecode_context_destroy(bc_ctx);
}
backwards compatibility for target type 13 json scanning
2015-03-04 12:08:34 -05:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/* backwards compatibility: scan the json string unless a virus was detected */
Remove PCRE dead code As of ClamAV 0.105, PCRE2 is required. PCRE (1) is not an option, and there is also no option to disable PCRE support. This commit removes the dead code associated with those old build options.
2024-03-25 10:32:38 -04:00
if (status != CL_VIRUS && (iroot->ac_lsigs || iroot->ac_patterns || iroot->pcre_metas)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_dbgmsg("scan_common: running deprecated preclass bytecodes for target type 13\n");
ctx.options->general &= ~CL_SCAN_GENERAL_COLLECT_METADATA;
status = cli_magic_scan_buff(jstring, strlen(jstring), &ctx, NULL, LAYER_ATTRIBUTES_NONE);
Add callback for user processing of json string and result of json scan.
2014-06-03 13:31:50 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
Only scan json props file if a virus is not found in the initial scan. Add --leave-temps support for json props file.
2014-05-05 17:05:26 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
* Invoke file props callback.
*/
if (ctx.engine->cb_file_props != NULL) {
ret = ctx.engine->cb_file_props(jstring, status, ctx.cb_ctx);
if (ret != CL_SUCCESS) {
status = ret;
Add callback for user processing of json string and result of json scan.
2014-06-03 13:31:50 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
Add callback for user processing of json string and result of json scan.
2014-06-03 13:31:50 -04:00
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
/*
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
* Write the file properties metadata JSON to metadata.json if keeptmp is enabled and temp-dir recursion is enabled.
* At present, the `metadata.json` filename is hardcoded, and cannot be written to a directory containing temp files from other scans.
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
*/
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ctx.engine->keeptmp) &&
(ctx.engine->engine_options & ENGINE_OPTIONS_TMPDIR_RECURSION)) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
int fd = -1;
char *tmpname = NULL;
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
if ((ret = cli_newfilepathfd(ctx.this_layer_tmpdir, "metadata.json", &tmpname, &fd)) != CL_SUCCESS) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_dbgmsg("scan_common: Can't create json properties file, ret = %i.\n", ret);
} else {
if ((size_t)-1 == cli_writen(fd, jstring, strlen(jstring))) {
cli_dbgmsg("scan_common: cli_writen error writing json properties file.\n");
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
} else {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_dbgmsg("json written to: %s\n", tmpname);
Only scan json props file if a virus is not found in the initial scan. Add --leave-temps support for json props file.
2014-05-05 17:05:26 -04:00
}
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
if (fd != -1) {
close(fd);
}
if (NULL != tmpname) {
free(tmpname);
Only scan json props file if a virus is not found in the initial scan. Add --leave-temps support for json props file.
2014-05-05 17:05:26 -04:00
}
}
Add build support for libjson and initial json work for properties file.
2014-04-16 16:40:56 -04:00
}
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
// If any alerts occurred, set the output pointer to the "latest" alert signature name.
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
if (0 < evidence_num_alerts(ctx.this_layer_evidence)) {
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
*last_alert_out = cli_get_last_virus_str(&ctx);
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
}
libclamav: Fix issue reporting trusted verdicts If the outermost layer is trusted (e.g. using an FP signature), the verdict passed back by the `cl_scan*_ex()` functions should be CL_VERDICT_TRUSTED. To ensure this, and other correct verdicts, I moved the logic setting the verdict to occur when adding indicators, or trusting a given layer. Then at the end of a scan, it will set the output verdict parameter to the top level verdict. This commit also: * Fixes a bug in the `ex_scan_callbacks` program where a crash would happen when a hash was retrieved for an inner layer, but isn't for the container. * Added debug logs whenever a hash is calculated or set, printing the hash type and hash string. * When a layer is trusted, in addition to removing evidence for that layer, it will also index the metadata JSON (if that feature is enabled) and will rename any "Viruses" to "IgnoredAlerts", and rename "ContainedIndicators" to "IgnoredContainedIndicators". * Fixed an issue where setting the hash algorithm with extra characters, like setting to "sha256789" would ignore the extra characters, and report the hash type as the same. It will now fail if the string length differs from the known hash algorithm.
2025-08-07 13:22:57 -04:00
*verdict_out = ctx.recursion_stack[ctx.recursion_level].verdict;
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
/*
* Report PUA alerts here.
*/
num_potentially_unwanted_indicators = evidence_num_indicators_type(
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
ctx.this_layer_evidence,
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
IndicatorType_PotentiallyUnwanted);
if (0 != num_potentially_unwanted_indicators) {
// We have "potentially unwanted" indicators that would not have been reported yet.
// We may wish to report them now, ... depending ....
if (ctx.options->general & CL_SCAN_GENERAL_ALLMATCHES) {
// We're in allmatch mode, so report all "potentially unwanted" matches now.
size_t i;
for (i = 0; i < num_potentially_unwanted_indicators; i++) {
const char *pua_alert = evidence_get_indicator(
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
ctx.this_layer_evidence,
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
IndicatorType_PotentiallyUnwanted,
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
i,
NULL, // Don't need to get the depth here.
NULL // Don't need to get the object ID here.
);
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
if (NULL != pua_alert) {
// We don't know exactly which layer the alert happened at.
// There's a decent chance it wasn't at this layer, and in that case we wouldn't
// even have access to that file anymore (it's gone!). So we'll pass back -1 for the
// file descriptor rather than using `cli_virus_found_cb() which would pass back
// The top level file descriptor.
if (ctx.engine->cb_virus_found) {
ctx.engine->cb_virus_found(
-1,
pua_alert,
ctx.cb_ctx);
}
}
}
} else {
// Not allmatch mode. Only want to report one thing...
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
if (0 == evidence_num_indicators_type(ctx.this_layer_evidence, IndicatorType_Strong)) {
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
// And it looks like we haven't reported anything else, so report the last "potentially unwanted" one.
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
// cli_get_last_virus() will do that, grabbing the last alerting indicator of any type.
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
cl_error_t callback_ret = CL_SUCCESS;
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
while ((CL_SUCCESS == callback_ret) &&
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
(0 < evidence_num_indicators_type(ctx.this_layer_evidence, IndicatorType_PotentiallyUnwanted))) {
callback_ret = cli_virus_found_cb(
&ctx,
cli_get_last_virus(&ctx),
IndicatorType_PotentiallyUnwanted);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
// If the callback returned CL_SUCCESS then it will have also removed the indicator from evidence
libclamav: scan-layer callback API functions Add the following scan callbacks: ```c cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH); cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN); cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN); cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT); cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE); ``` Each callback may alter scan behavior using the following return codes: * CL_BREAK Scan aborted by callback (the rest of the scan is skipped). This does not mark the file as clean or infected, it just skips the rest of the scan. * CL_SUCCESS / CL_CLEAN File scan will continue. This is different than CL_VERIFIED because it does not affect prior or future alerts. Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip the rest of the scan for this layer. * CL_VIRUS This means you don't trust the file. A new alert will be added. For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed). * CL_VERIFIED Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer. You might want to do this if you trust the hash or verified a digital signature. The rest of the scan will be skipped FOR THIS layer. For contained files, this does NOT mean that the parent or adjacent layers are trusted. Each callback is given a pointer to the current scan layer from which they can get previous layers, can get the the layer's fmap, and then various attributes of the layer and of the fmap such as: - layer recursion level - layer object id - layer file type - layer attributes (was decerypted, normalized, embedded, or re-typed) - layer last alert - fmap name - fmap hash (md5, sha1, or sha2-256) - fmap data (pointer and size) - fmap file descriptor, if any (fd, offset, size) - fmap filepath, if any (filepath, offset, size) To make this possible, this commits introduced a handful of new APIs to query scan-layer details and fmap details: - `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);` - `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);` - `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);` - `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);` - `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);` - `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);` - `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);` - `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);` - `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);` - `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);` - `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);` - `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);` - `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);` - `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);` - `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);` - `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);` - `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);` This commit deprecates but does not remove the existing scan callbacks: - `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);` - `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);` - `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);` - `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);` - `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);` - `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);` This commit also adds an interactive test program to demonstrate the callbacks. See: `examples/ex_scan_callbacks.c` CLAM-255 CLAM-2485 CLAM-2626
2025-06-22 14:37:03 -04:00
// And we must loop around and report the next one.
}
Fix allmatch issue with preclass bytecode The verdict is being recorded before the preclass bytecode hook meaning that the final verdict may come back as "clean" in allmatch mode, even if the preclass bytecode hook matches something. This commit moves the verdict check to occur AFTER the preclass bytecode hook executes.
2023-02-02 19:00:37 -08:00
}
}
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
/*
* If the caller requested a hash, we need to get it from the fmap.
*/
if (NULL != hash_out) {
// Allocate a buffer for the hash
size_t hash_len = cli_hash_len(requested_hash_type);
char *hash_string = malloc(hash_len * 2 + 1); // +1 for the null terminator
if (NULL == hash_string) {
cli_errmsg("scan_common: no memory for hash string buffer\n");
status = CL_EMEM;
} else {
// Get the hash from the fmap.
uint8_t *hash = NULL;
ret = fmap_get_hash(map, &hash, requested_hash_type);
if (CL_SUCCESS != ret || hash == NULL) {
cli_errmsg("scan_common: fmap_get_hash failed: %d\n", ret);
status = ret;
} else {
// Convert hash to string.
size_t i;
for (i = 0; i < hash_len; i++) {
sprintf(hash_string + i * 2, "%02x", hash[i]);
}
hash_string[hash_len * 2] = 0;
*hash_out = hash_string;
}
}
}
/*
* If the caller requested a file type, we need to get it from the fmap.
*/
if (NULL != file_type_out) {
const char *ftname = cli_ftname(ctx.recursion_stack[ctx.recursion_level].type);
if ((NULL == ftname) ||
(strcmp(ftname, "CL_TYPE_ANY") == 0)) {
cli_dbgmsg("scan_common: unknown file type.\n");
// Default to CL_TYPE_BINARY_DATA if we never determined the type.
*file_type_out = cli_safer_strdup("CL_TYPE_BINARY_DATA");
} else {
// Set the output pointer to the file type name.
*file_type_out = cli_safer_strdup(ftname);
}
}
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
done:
Do not report timeout as error Because of an oversight, the `ctx->abort_scan` flag was propagating timeout and perhaps other error types to the final verdict. Fixed this in the `result_should_goto_done()` function. Some non-fatal errors (notably CL_ETIMEOUT) may appear between the call to the magic-scan, and the end of scan_common(). We need to filter these status results at this level as well to prevent this from happening. I also fixed a warning accidentally introduced earlier caused by trying to use a string after free'ing it and setting it to NULL. And I explicitly set the status to CL_SUCCESS if the cache check comes back clean, rather than trusting that no one had messed with the `ret` variable since it was let set (to CL_SUCCESS). It's more readable and less prone to future bugs.
2022-09-21 12:22:54 -07:00
Fix typos (no functional changes)
2024-01-19 09:08:36 -08:00
if (logg_initialized) {
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
cli_logg_unsetup();
}
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (NULL != ctx.metadata_json) {
cli_json_delobj(ctx.metadata_json);
Scanners: Remove allmatch checks + significant code cleanup Also fixed a number of conditions where magic_scan() critical errors may be ignored. To ensure that the scan truly aborts for signature matches (not in allmatch mode) and for timeouts, the `ctx->abort` option is now set in these two conditions, and checked in several spots in magic_scan(). Additionally, I've consolidated some of the "scan must halt" type of checks (mostly large switch statements) into a function so that we can use the exact same logic in a few places in magic_scan(). I've also fixed a few minor warnings and code format issues.
2022-08-27 10:46:21 -07:00
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
if ((NULL != ctx.engine) &&
(ctx.engine->engine_options & ENGINE_OPTIONS_TMPDIR_RECURSION) &&
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
(NULL != ctx.this_layer_tmpdir)) {
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
if (!ctx.engine->keeptmp) {
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
(void)cli_rmdirs(ctx.this_layer_tmpdir);
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
libclamav: Add engine option to toggle temp directory recursion Temp directory recursion in ClamAV is when each layer of a scan gets its own temp directory in the parent layer's temp directory. In addition to temp directory recursion, ClamAV has been creating a new subdirectory for each file scan as a risk-adverse method to ensure no temporary file leaks fill up the disk. Creating a directory is relatively slow on Windows in particular if scanning a lot of very small files. This commit: 1. Separates the temp directory recursion feature from the leave-temps feature so that libclamav can leave temp files without making subdirectories for each file scanned. 2. Makes it so that when temp directory recursion is off, libclamav will just use the configure temp directory for all files. The new option to enable temp directory recursion is for libclamav-only at this time. It is off by default, and you can enable it like this: ```c cl_engine_set_num(engine, CL_ENGINE_TMPDIR_RECURSION, 1); ``` For the `clamscan` and `clamd` programs, temp directory recursion will be enabled when `--leave-temps` / `LeaveTemporaryFiles` is enabled. The difference is that when disabled, it will return to using the configured temp directory without making a subdirectory for each file scanned, so as to improve scan performance for small files, mostly on Windows. Under the hood, this commit also: 1. Cleans up how we keep track of tmpdirs for each layer. The goal here is to align how we keep track of layer-specific stuff using the scan_layer structure. 2. Cleans up how we record metadata JSON for embedded files. Note: Embedded files being different from Contained files, as they are extracted not with a parser, but by finding them with file type magic signatures. CLAM-1583
2025-06-09 20:42:31 -04:00
free(ctx.this_layer_tmpdir);
} else {
// If we didn't create a temp directory, we don't need to free it,
// and have to trust that all temp files were cleaned up by their respective modules.
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
}
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
if (NULL != target_basename) {
free(target_basename);
}
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
if (NULL != ctx.target_filepath) {
free(ctx.target_filepath);
}
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
if (NULL != ctx.perf) {
perf_done(&ctx);
}
if (NULL != ctx.hook_lsig_matches) {
cli_bitset_free(ctx.hook_lsig_matches);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
if (NULL != ctx.recursion_stack) {
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
if (NULL != ctx.recursion_stack[ctx.recursion_level].evidence) {
evidence_free(ctx.recursion_stack[ctx.recursion_level].evidence);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
free(ctx.recursion_stack);
Fix several coverity warnings 290424 Missing break in switch - In hash_match: Missing break statement between cases in switch statement 290414 Resource leak - In cli_scanishield_msi: Leak of memory or pointers to system resources. Memory leak in a fail case 288197 Resource leak - In decrypt_any: Leak of memory or pointers to system resources. Memory leak in a fail case 290426 Resource leak - In cli_magic_scan: Leak of memory or pointers to system resources. Leaked a file prefix when running with --save-temps 192923 Resource leak - In cli_scanrar: Leak of memory or pointers to system resources. Leaked a file descriptor if a virus was found in a RAR file comment 225146 Resource leak - In cli_scanegg: Leak of memory or pointers to system resources. Leaked a file descriptor if unable to write a comment file to disk 290425 Resource leak - In scan_common: Leak of memory or pointers to system resources. Memory leaks in various fail cases. Also changes cli_scanrar to write out the file comment only if --leave-temps is specified and scan the buffer (like what is done in cli_scanegg) instead of writing the file out, scanning that, and then deleting the file if --leave-temps is not specified. The unit tests stopped working when correcting an issue with a switch statement that determined what type of signature had matched on a Google SafeBrowsing GDB rule. Looking into the unit tests, it looks like the code had always assumed that the test cases would be detected by a malware test rule in unit_tests/input/daily.gdb, but now some of the tests get matched on the phishing test rule. I updated the test logic to be more clear, and added tests for both cases now. Fix some memory leaks in libclamav/scanners.c
2020-07-15 08:39:32 -07:00
}
if (NULL != ctx.options) {
free(ctx.options);
}
New tmp sub-dir per scan; JSON meta improvements This commit improves the layout of the tmp file output and the JSON metadata output when using the --leave-temps and --gen-json options. For all scans, each scan target will get a unique tmp sub-directory. If using --leave-temps, that subdir will include the basename of the original file to make it easier to identify. Additionally, when using --leave-temps option, all extracted objects will have their subdirectories extracted in recursive subdirectories including filename prefixes where available. When not using the --leave-temps option, the layout of the tmp sub-directory will remain flat, so as to alleviate the possibility of exceeding PATH_MAX. The JSON metadata generated by the --gen-json option is now generated for all file types, not just a select few. The format is also pretty-printed for readability and now includes filenames and file paths when available. Also: - Added missing ALLMATCH check when determining if bytecode hooks should be run. - Added cl_engine_get_str API to windows libclamav symbol export file.
2019-05-23 22:50:04 -04:00
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
return status;
clamav-for-windows draft#2
2010-07-07 03:01:55 +02:00
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
cl_error_t cl_scandesc(
int desc,
const char *filename,
const char **virname,
unsigned long int *scanned,
const struct cl_engine *engine,
struct cl_scan_options *scanoptions)
{
cl_error_t status;
uint64_t scanned_out;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_verdict_t verdict_out = CL_VERDICT_NOTHING_FOUND;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
status = cl_scandesc_ex(
desc,
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&verdict_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
virname,
&scanned_out,
engine,
scanoptions,
NULL,
NULL,
NULL,
NULL,
NULL,
NULL);
if (NULL != scanned) {
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
if ((SIZEOF_LONG == 4) &&
(scanned_out / CL_COUNT_PRECISION > UINT32_MAX)) {
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
cli_warnmsg("cl_scanfile_callback: scanned_out exceeds UINT32_MAX, setting to UINT32_MAX\n");
*scanned = UINT32_MAX;
} else {
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
*scanned = (unsigned long int)(scanned_out / CL_COUNT_PRECISION);
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
}
}
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (verdict_out == CL_VERDICT_STRONG_INDICATOR || verdict_out == CL_VERDICT_POTENTIALLY_UNWANTED) {
// Reporting "CL_VIRUS" is more important than reporting an error,
// because... unfortunately we can only do one with this API.
status = CL_VIRUS;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
return status;
}
cl_error_t cl_scandesc_callback(
int desc,
const char *filename,
const char **virname,
unsigned long int *scanned,
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context)
{
cl_error_t status;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
uint64_t scanned_bytes;
cl_verdict_t verdict_out = CL_VERDICT_NOTHING_FOUND;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
status = cl_scandesc_ex(
desc,
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&verdict_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
virname,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&scanned_bytes,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
engine,
scanoptions,
context,
NULL,
NULL,
NULL,
NULL,
NULL);
if (NULL != scanned) {
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
if ((SIZEOF_LONG == 4) &&
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
(scanned_bytes / CL_COUNT_PRECISION > UINT32_MAX)) {
cli_warnmsg("cl_scanfile_callback: scanned_bytes exceeds UINT32_MAX, setting to UINT32_MAX\n");
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
*scanned = UINT32_MAX;
} else {
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
*scanned = (unsigned long int)(scanned_bytes / CL_COUNT_PRECISION);
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
}
}
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (verdict_out == CL_VERDICT_STRONG_INDICATOR || verdict_out == CL_VERDICT_POTENTIALLY_UNWANTED) {
// Reporting "CL_VIRUS" is more important than reporting an error,
// because... unfortunately we can only do one with this API.
status = CL_VIRUS;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
return status;
}
cl_error_t cl_scandesc_ex(
int desc,
const char *filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_verdict_t *verdict_out,
const char **last_alert_out,
uint64_t *scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context,
const char *hash_hint,
char **hash_out,
const char *hash_alg,
const char *file_type_hint,
char **file_type_out)
fmap scan API
2011-06-14 21:49:39 +03:00
{
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
cl_error_t status = CL_SUCCESS;
cl_fmap_t *map = NULL;
STATBUF sb;
char *filename_base = NULL;
if (FSTAT(desc, &sb) == -1) {
cli_errmsg("cl_scandesc_callback: Can't fstat descriptor %d\n", desc);
status = CL_ESTAT;
goto done;
}
if (sb.st_size <= 5) {
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
cli_dbgmsg("cl_scandesc_callback: File too small (" STDu64 " bytes), ignoring\n", (uint64_t)sb.st_size);
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
goto done;
}
Fix ability to disable filesize limit with libclamav C API You should be able to disable the maxfilesize limit by setting it to zero. When "disabled", ClamAV should defer to inherent limitations, which at this time is INT_MAX - 2 bytes. This works okay for ClamScan and ClamD because our option parser converts max-filesize=0 to 4294967295 (4GB). But it is presently broken for other applications using the libclamav C API, like this: ```c cl_engine_set_num(engine, CL_ENGINE_MAX_FILESIZE, 0); ``` The limit checks added for cl_scanmap_callback and cl_scanfile_callback in 0.103.4 and 0.104.1 broke this ability because we forgot to check if the `maxfilesize > 0` before enforcing it. This commit adds that guard so you can disable by setting to `0`. While working on this, I also found that the `max_size` variables in our libmspack scanner code are using an `off_t` type, which is a SIGNED integer that may be 32bit width even on some 64bit platforms, or may be a 64bit width. AND the default `max_size` when `maxfilesize == 0` was being set to UINT_MAX (0xffffffff), aka `-1` when `off_t` is 32bits. This commit addresses this related issue by: - changing the `max_size` to use `uint64_t`, like our other limits. - verifying that `maxfilesize > 0` before using it. - checking that using `UINT32_MAX` as a backup will not exceed the max-scansize in the same way that we do with the maxfilesize.
2021-12-22 13:29:18 -08:00
if ((engine->maxfilesize > 0) && ((uint64_t)sb.st_size > engine->maxfilesize)) {
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
cli_dbgmsg("cl_scandesc_callback: File too large (" STDu64 " bytes), ignoring\n", (uint64_t)sb.st_size);
if (scanoptions->heuristic & CL_SCAN_HEURISTIC_EXCEEDS_MAX) {
ClamD: Fix error reporting when exceeding max filesize Previously clamd would report "Cannot allocate memory" when a file exceeded max file size. This commit corrects it to report "Heuristics.Limits.Exceeded.MaxFileSize." Fixes: https://github.com/Cisco-Talos/clamav/issues/670.
2023-08-17 10:44:26 -07:00
if (engine->cb_virus_found) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
engine->cb_virus_found(desc, "Heuristics.Limits.Exceeded.MaxFileSize", context);
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (last_alert_out) {
*last_alert_out = "Heuristics.Limits.Exceeded.MaxFileSize";
ClamD: Fix error reporting when exceeding max filesize Previously clamd would report "Cannot allocate memory" when a file exceeded max file size. This commit corrects it to report "Heuristics.Limits.Exceeded.MaxFileSize." Fixes: https://github.com/Cisco-Talos/clamav/issues/670.
2023-08-17 10:44:26 -07:00
}
}
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
status = CL_VIRUS;
} else {
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
status = CL_SUCCESS;
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
}
goto done;
}
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
if (NULL != filename) {
Windows: Fix filepath basename issue On Windows, the cli_basename function should treat both '/' and '\' as path separators. Most Windows APIs also accept both. On Linux/Unix, it makes sense when using a filepath that is more for informational purposes or where it may have come from a Windows system, to treat the '\' as a path separator. But in situations where the the path is needed for some critical action, like moving or deleting a file, we can't treat it as a path separator.
2025-08-11 18:02:09 -04:00
(void)cli_basename(filename, strlen(filename), &filename_base, true /* posix_support_backslash_pathsep */);
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
}
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
if (NULL == (map = fmap_new(desc, 0, sb.st_size, filename_base, filename))) {
cli_errmsg("CRITICAL: fmap_new() failed\n");
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
status = CL_EMEM;
goto done;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
status = scan_common(
map,
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
verdict_out,
last_alert_out,
scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
engine,
scanoptions,
context,
hash_hint,
hash_out,
hash_alg,
file_type_hint,
file_type_out);
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
done:
if (NULL != map) {
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(map);
bb12506: Fix phishing/heuristic alert verbosity Some detections, like phishing, are considered heuristic alerts because they match based on behavior more than on content. A subset of these are considered "potentially unwanted" (low-severity). These low-severity alerts include: - phishing - PDFs with obfuscated object names - bytecode signature alerts that start with "BC.Heuristics" The concept is that unless you enable "heuristic precedence" (a method of lowing the threshold to immediateley alert on low-severity detections), the scan should continue after a match in case a higher severity match is found. Only at the end will it print the low-severity match if nothing else was found. The current implementation is buggy though. Scanning of archives does not correctly bail out for the entire archive if one email contains a phishing link. Instead, it sets the "heuristic found" flag then and alerts for every subsequent file in the archive because it doesn't know if the heuristic was found in an embedded file or the target file. Because it's just a heuristic and the status is "clean", it keeps scanning. This patch corrects the behavior by checking if a low-severity alerts were found at the end of scanning the target file, instead of at the end of each embedded file. Additionally, this patch fixes an in issue with phishing alerts wherein heuristic precedence mode did not cause a scan to stop after the first alert. The above changes required restructuring to create an fmap inside of cl_scandesc_callback() so that scan_common() could be modified to require an fmap and set up so that the current *ctx->fmap pointer is never NULL when scan_common() evaluates match results. Also fixed a couple minor bugs in the phishing unit tests and cleaned up the test code for improved legitibility and type safety.
2020-04-18 10:46:57 -04:00
}
if (NULL != filename_base) {
free(filename_base);
}
return status;
fmap scan API
2011-06-14 21:49:39 +03:00
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
cl_error_t cl_scanmap_callback(
cl_fmap_t *map,
const char *filename,
const char **virname,
unsigned long int *scanned,
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context)
{
cl_error_t status;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
uint64_t scanned_bytes;
cl_verdict_t verdict_out = CL_VERDICT_NOTHING_FOUND;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
status = cl_scanmap_ex(
map,
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&verdict_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
virname,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&scanned_bytes,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
engine,
scanoptions,
context,
NULL,
NULL,
NULL,
NULL,
NULL);
if (NULL != scanned) {
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read The ClamScan scan summary prints bytes scanned and bytes read in multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the `cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and `cl_scandesc_callback()` functions. I believe this imprecision was the result of using an `unsigned long int` which may be 64bit or 32bit, depending on platform. I believe the intention was to be able to support scanning more than 4 GiB of data. Since the new `cl_scan*_ex()` functions use a `uint64_t`, which guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no reason not to report an accurate count. For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION` behavior to maintain backwards compatibility. I have also improved the bytes scanned/read output to report GiB, MiB, KiB, or B as appropriate. Previously, it always report "MB". CLAM-1433
2025-06-25 14:39:11 -04:00
if ((SIZEOF_LONG == 4) &&
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
(scanned_bytes / CL_COUNT_PRECISION > UINT32_MAX)) {
cli_warnmsg("cl_scanfile_callback: scanned_bytes exceeds UINT32_MAX, setting to UINT32_MAX\n");
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
*scanned = UINT32_MAX;
} else {
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
*scanned = (unsigned long int)(scanned_bytes / CL_COUNT_PRECISION);
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
}
}
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (verdict_out == CL_VERDICT_STRONG_INDICATOR || verdict_out == CL_VERDICT_POTENTIALLY_UNWANTED) {
// Reporting "CL_VIRUS" is more important than reporting an error,
// because... unfortunately we can only do one with this API.
status = CL_VIRUS;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
return status;
}
cl_error_t cl_scanmap_ex(
cl_fmap_t *map,
const char *filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_verdict_t *verdict_out,
const char **last_alert_out,
uint64_t *scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context,
const char *hash_hint,
char **hash_out,
const char *hash_alg,
const char *file_type_hint,
char **file_type_out)
fmap scan API
2011-06-14 21:49:39 +03:00
{
Fix ability to disable filesize limit with libclamav C API You should be able to disable the maxfilesize limit by setting it to zero. When "disabled", ClamAV should defer to inherent limitations, which at this time is INT_MAX - 2 bytes. This works okay for ClamScan and ClamD because our option parser converts max-filesize=0 to 4294967295 (4GB). But it is presently broken for other applications using the libclamav C API, like this: ```c cl_engine_set_num(engine, CL_ENGINE_MAX_FILESIZE, 0); ``` The limit checks added for cl_scanmap_callback and cl_scanfile_callback in 0.103.4 and 0.104.1 broke this ability because we forgot to check if the `maxfilesize > 0` before enforcing it. This commit adds that guard so you can disable by setting to `0`. While working on this, I also found that the `max_size` variables in our libmspack scanner code are using an `off_t` type, which is a SIGNED integer that may be 32bit width even on some 64bit platforms, or may be a 64bit width. AND the default `max_size` when `maxfilesize == 0` was being set to UINT_MAX (0xffffffff), aka `-1` when `off_t` is 32bits. This commit addresses this related issue by: - changing the `max_size` to use `uint64_t`, like our other limits. - verifying that `maxfilesize > 0` before using it. - checking that using `UINT32_MAX` as a backup will not exceed the max-scansize in the same way that we do with the maxfilesize.
2021-12-22 13:29:18 -08:00
if ((engine->maxfilesize > 0) && (map->len > engine->maxfilesize)) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cli_dbgmsg("cl_scandesc_callback: File too large (%zu bytes), ignoring\n", map->len);
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
if (scanoptions->heuristic & CL_SCAN_HEURISTIC_EXCEEDS_MAX) {
Error reporting touchup for max-filesize
2023-08-23 15:26:50 -07:00
if (engine->cb_virus_found) {
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
engine->cb_virus_found(fmap_fd(map), "Heuristics.Limits.Exceeded.MaxFileSize", context);
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (last_alert_out) {
*last_alert_out = "Heuristics.Limits.Exceeded.MaxFileSize";
Error reporting touchup for max-filesize
2023-08-23 15:26:50 -07:00
}
}
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
return CL_VIRUS;
}
Replace CL_CLEAN with CL_SUCCESS in clamav.h, and scanners.c Both enum variants are 0, so it's a no-op. `cl_error_t` should be used to determine if we keep going or stop, whether that's because there was a detection and we're not in allmatch mode, or if because of an error. `cl_verdict_t` should be used to determine or say the verdict (clean is a verdict, though I feel 'nothing found' is more accurate).
2025-07-29 10:44:13 -04:00
return CL_SUCCESS;
Fix errors when scanning files > 4G This commit resolves https://bugzilla.clamav.net/show_bug.cgi?id=12673 Changes in 0.103 to order of operations for creating fmaps and performaing hashes of fmaps resulted errors when scanning files that are 4096M and a different (but related) error when scanning files > 4096M. This is despite the fact that scanning is supposed to be limited to --max-scansize (MaxScanSize) and was also apparently limited to INT_MAX - 2 (aka ~1.999999G) back in 2014 to alleviate reported crashes for a few large file formats. (see https://bugzilla.clamav.net/show_bug.cgi?id=10960) This last limitation was not documented, so I added it to the sample clamd.conf. Anyways, the main issue is that the fmap module was using "unsigned int" and was then enforcing a limitation (verbose error messages) when that a map length exceeded the capapacity of an unsigned int. This commit switches the associated variables over to uint64_t, and while fmaps are still limited to size_t in other places, the fmap module will at least work with files > 4G on 64bit systems. In testing this, I found that the time to hash a file, particularly when hashing a file on an NTFS partition from Linux was really slow because we were hashing in FILEBUFF chunks (about 8K) at a time. Increasing this to 10MB chunks speeds up scanning of large files. Finally, now that hashing is performed immediately when an fmap is created for a file, hashing of files larger than max-scansize was occuring. This commit adds checks to bail out early if the file size exceeds the maximum before creating an fmap. It will alert with the Heuristics.Limits.Exceeded name if the heuristic is enabled. Also fixed CheckFmapFeatures.cmake module that detects if sysconf(_SC_PAGESIZE) is available.
2021-03-31 12:16:41 -07:00
}
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
if (NULL != filename && map->name == NULL) {
// Use the provided name for the fmap name if one wasn't already set.
Windows: Fix filepath basename issue On Windows, the cli_basename function should treat both '/' and '\' as path separators. Most Windows APIs also accept both. On Linux/Unix, it makes sense when using a filepath that is more for informational purposes or where it may have come from a Windows system, to treat the '\' as a path separator. But in situations where the the path is needed for some critical action, like moving or deleting a file, we can't treat it as a path separator.
2025-08-11 18:02:09 -04:00
cli_basename(filename, strlen(filename), &map->name, true /* posix_support_backslash_pathsep */);
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
return scan_common(
map,
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
verdict_out,
last_alert_out,
scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
engine,
scanoptions,
context,
hash_hint,
hash_out,
hash_alg,
file_type_hint,
file_type_out);
fmap scan API
2011-06-14 21:49:39 +03:00
}
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
cl_error_t cli_magic_scan_file(const char *filename, cli_ctx *ctx, const char *original_name, uint32_t attributes)
bugfixes git-svn: trunk@492
2004-04-14 22:55:44 +00:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int fd = -1;
cl_error_t ret = CL_EOPEN;
bugfixes git-svn: trunk@492
2004-04-14 22:55:44 +00:00
do not scan mail files twice; separate archive and mail recursion limits git-svn: trunk@650
2004-07-04 14:56:48 +00:00
/* internal version of cl_scanfile with arec/mrec preserved */
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
fd = safe_open(filename, O_RDONLY | O_BINARY);
if (fd < 0) {
goto done;
}
bugfixes git-svn: trunk@492
2004-04-14 22:55:44 +00:00
Code quality: Refactor layer attributes as scan parameter The current implementation sets a "next layer attributes" flag field in the scan context. This may introduce bugs if accidentally not cleared during error handling, causing that attribute to be applied to a different layer than intended. This commit resolves that by adding an attribute flag to the major internal scan functions and removing the "next layer attributes" from the scan context. This attributes flag shares the same flag fields as the attributes flag in the new file inspection callback and the flags are defined in `clamav.h`.
2022-03-09 22:26:40 -08:00
ret = cli_magic_scan_desc(fd, filename, ctx, original_name, attributes);
bugfixes git-svn: trunk@492
2004-04-14 22:55:44 +00:00
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
done:
if (fd >= 0) {
close(fd);
}
Adds file inspection callback and example code libclamav callbacks can be used to access embedded file content at each layer of extraction during the course of a scan. The existing callbacks only provide access to the file descriptor and a guess at the file type. This patch adds a new callback for the purposes of file/archive inspection that provides additional insight into the embedded file. This includes: - ancestors: an array of parent file names - parent file size: the size of the direct parent layer - file name: current layer's filename, if any. - file buffer (pointer) - file size: size of file buffer - file type: just a guess at the current file's type - file descriptor: may be -1 if the layer is in-memory only. - layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h Two new example apps are added that are automatically built when compiling under CMake: - ex2 demonstrates the prescan callback. - ex3 demonstrates the new file inspection callback. The examples are now installed if enabled, so you can test them in the Docker image, and so that they'll be colocated with the DLLs so you can test them on Windows. The installed examples should also be able to find the UnRAR library at run time, without having to set LD_LIBRARY_PATH. This commit also sets the fmap->name in an fmap-scan using the basname of the provided filename if the caller provided the filename and the provided fmap does not have the name set.
2020-12-18 09:13:17 -08:00
bugfixes git-svn: trunk@492
2004-04-14 22:55:44 +00:00
return ret;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
cl_error_t cl_scanfile(
const char *filename,
const char **virname,
unsigned long int *scanned,
const struct cl_engine *engine,
struct cl_scan_options *scanoptions)
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
{
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
cl_error_t status;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
uint64_t scanned_bytes;
cl_verdict_t verdict_out = CL_VERDICT_NOTHING_FOUND;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
status = cl_scanfile_ex(
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&verdict_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
virname,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&scanned_bytes,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
engine,
scanoptions,
NULL,
NULL,
NULL,
NULL,
NULL,
NULL);
if (NULL != scanned) {
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (SIZEOF_LONG == 4 && scanned_bytes > UINT32_MAX) {
cli_warnmsg("cl_scanfile_callback: scanned_bytes exceeds UINT32_MAX, setting to UINT32_MAX\n");
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
*scanned = UINT32_MAX;
} else {
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
*scanned = (unsigned long int)scanned_bytes;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
}
}
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (verdict_out == CL_VERDICT_STRONG_INDICATOR || verdict_out == CL_VERDICT_POTENTIALLY_UNWANTED) {
// Reporting "CL_VIRUS" is more important than reporting an error,
// because... unfortunately we can only do one with this API.
status = CL_VIRUS;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
return status;
}
cl_error_t cl_scanfile_callback(
const char *filename,
const char **virname,
unsigned long int *scanned,
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context)
{
cl_error_t status;
uint64_t scanned_out;
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_verdict_t verdict_out = CL_VERDICT_NOTHING_FOUND;
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
status = cl_scanfile_ex(
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
&verdict_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
virname,
&scanned_out,
engine,
scanoptions,
context,
NULL,
NULL,
NULL,
NULL,
NULL);
if (NULL != scanned) {
if (SIZEOF_LONG == 4 && scanned_out > UINT32_MAX) {
cli_warnmsg("cl_scanfile_callback: scanned_out exceeds UINT32_MAX, setting to UINT32_MAX\n");
*scanned = UINT32_MAX;
} else {
*scanned = (unsigned long int)scanned_out;
}
}
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
if (verdict_out == CL_VERDICT_STRONG_INDICATOR || verdict_out == CL_VERDICT_POTENTIALLY_UNWANTED) {
// Reporting "CL_VIRUS" is more important than reporting an error,
// because... unfortunately we can only do one with this API.
status = CL_VIRUS;
}
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
return status;
Initial revision git-svn: trunk@7
2003-07-29 15:48:06 +00:00
}
improve handling of PDF, CAB, RTF, OLE2 and HTML files (sync with branch/0.93) git-svn: trunk@3862
2008-05-27 16:30:47 +00:00
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
cl_error_t cl_scanfile_ex(
const char *filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
cl_verdict_t *verdict_out,
const char **last_alert_out,
uint64_t *scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
const struct cl_engine *engine,
struct cl_scan_options *scanoptions,
void *context,
const char *hash_hint,
char **hash_out,
const char *hash_alg,
const char *file_type_hint,
char **file_type_out)
clamd: add ExtendedDetectionInfo (bb#1228, #1626)
2010-07-06 19:46:55 +02:00
{
Record names of extracted files A way is needed to record scanned file names for two purposes: 1. File names (and extensions) must be stored in the json metadata properties recorded when using the --gen-json clamscan option. Future work may use this to compare file extensions with detected file types. 2. File names are useful when interpretting tmp directory output when using the --leave-temps option. This commit enables file name retention for later use by storing file names in the fmap header structure, if a file name exists. To store the names in fmaps, an optional name argument has been added to any internal scan API's that create fmaps and every call to these APIs has been modified to pass a file name or NULL if a file name is not required. The zip and gpt parsers required some modification to record file names. The NSIS and XAR parsers fail to collect file names at all and will require future work to support file name extraction. Also: - Added recursive extraction to the tmp directory when the --leave-temps option is enabled. When not enabled, the tmp directory structure remains flat so as to prevent the likelihood of exceeding MAX_PATH. The current tmp directory is stored in the scan context. - Made the cli_scanfile() internal API non-static and added it to scanners.h so it would be accessible outside of scanners.c in order to remove code duplication within libmspack.c. - Added function comments to scanners.h and matcher.h - Converted a TDB-type macros and LSIG-type macros to enums for improved type safey. - Converted more return status variables from `int` to `cl_error_t` for improved type safety, and corrected ooxml file typing functions so they use `cli_file_t` exclusively rather than mixing types with `cl_error_t`. - Restructured the magic_scandesc() function to use goto's for error handling and removed the early_ret_from_magicscan() macro and magic_scandesc_cleanup() function. This makes the code easier to read and made it easier to add the recursive tmp directory cleanup to magic_scandesc(). - Corrected zip, egg, rar filename extraction issues. - Removed use of extra sub-directory layer for zip, egg, and rar file extraction. For Zip, this also involved changing the extracted filenames to be randomly generated rather than using the "zip.###" file name scheme.
2020-03-19 21:23:54 -04:00
int fd;
cl_error_t ret;
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
const char *fname = cli_to_utf8_maybe_alloc(filename);
clamd: add ExtendedDetectionInfo (bb#1228, #1626)
2010-07-06 19:46:55 +02:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (!fname)
return CL_EARG;
win32: UTF #6 - clamd proto
2011-04-18 17:25:23 +02:00
clamd: PR review fix, update acknowledgements The clamd TOCTOU access check fix introduced and expectation that the scanfile API will set errno if access was denied. We should instead use the cl_error_t error code enum. Also added Duane Waddle to the 0.104 contributors acknowledgements.
2021-01-21 16:32:54 -08:00
if ((fd = safe_open(fname, O_RDONLY | O_BINARY)) == -1) {
if (errno == EACCES) {
return CL_EACCES;
} else {
return CL_EOPEN;
}
}
clamd: add ExtendedDetectionInfo (bb#1228, #1626)
2010-07-06 19:46:55 +02:00
fixed minor warnings regarding type conversions.
2017-08-08 17:38:17 -04:00
if (fname != filename)
Updating libclamunrar from legacy C implementation to modern unrar 5.6.5. API changes and supporting changes included to pass the filepath of the scanned file into libclamav through the cli_ctx structure, required by the unrar library to open archives. The filename argument may be optional for the scandesc scanning variant, but libclamav will make a best effort to identify the filename from the file descriptor if it was not provided. In addition, included the ability to prefix temp file and directory names with file basenames.
2018-07-30 20:19:28 -04:00
free((char *)fname);
win32: UTF #6 - clamd proto
2011-04-18 17:25:23 +02:00
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
ret = cl_scandesc_ex(
fd,
filename,
libclamav: cl_scan*_ex() functions provide verdict separate from errors It is a shortcoming of existing scan APIs that it is not possible to return an error without masking a verdict. We presently work around this limitation by counting up detections at the end and then overriding the error code with `CL_VIRUS`, if necessary. The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions should provide the scan verdict separately from the error code. This introduces a new enum for recording and reporting a verdict: `cl_verdict_t` with options: - `CL_VERDICT_NOTHING_FOUND` - `CL_VERDICT_TRUSTED` - `CL_VERDICT_STRONG_INDICATOR` - `CL_VERDICT_POTENTIALLY_UNWANTED` Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED` if there is a (hash-based) FP signature for a file, or in the cause where Authenticode or similar certificate-based verification was performed, or in the case where an application scan callback returned `CL_VERIFIED`. CLAM-763 CLAM-865
2025-07-27 22:47:29 -04:00
verdict_out,
last_alert_out,
scanned_out,
libclamav: Add 'ex'-scan functions to API w. hash and type in/out parameters Add `cl_scanfile_ex()`, `cl_scanmap_ex()`, and `cl_scandesc_ex()` functions that provide the following additional parameters: hash_hint (Optional) A NULL terminated string of the file hash so that libclamav does not need to calculate it. [out] hash_out (Optional) A NULL terminated string of the file hash. The caller is responsible for freeing the string. hash_alg The hashing algorithm used for either `hash_hint` or `hash_out`. Supported algorithms are "md5", "sha1", "sha2-256". If not specified, the default is "sha2-256". file_type_hint (Optional) A NULL terminated string of the file type hint. E.g. "pe", "elf", "zip", etc. You may also use ClamAV type names such as "CL_TYPE_PE". ClamAV will ignore the hint if it is not familiar with the specified type. See also: https://docs.clamav.net/appendix/FileTypes.html#file-types file_type_out (Optional) A NULL terminated string of the file type of the top layer as determined by ClamAV. Will take the form of the standard ClamAV file type format. E.g. "CL_TYPE_PE". See also: https://docs.clamav.net/appendix/FileTypes.html#file-types CLAM-2626
2025-06-16 12:27:51 -04:00
engine,
scanoptions,
context,
hash_hint,
hash_out,
hash_alg,
file_type_hint,
file_type_out);
clamd: add ExtendedDetectionInfo (bb#1228, #1626)
2010-07-06 19:46:55 +02:00
close(fd);
return ret;
}
improve handling of PDF, CAB, RTF, OLE2 and HTML files (sync with branch/0.93) git-svn: trunk@3862
2008-05-27 16:30:47 +00:00
/*
Local Variables:
c-basic-offset: 4
End:
*/
Reference in a new issue Copy permalink