Stowage/clamav
2
0
Fork 0
mirror of https://github.com/Cisco-Talos/clamav.git synced 2025-10-19 18:33:16 +00:00
Code Issues Projects Releases Packages Wiki Activity
clamav/unit_tests/check_bytecode.c

654 lines
20 KiB
C
Raw Normal View History

Add missing unit-test files.
2009-07-13 19:45:05 +03:00
/*
Update copyright dates for 2020
2020-01-03 15:44:07 -05:00
* Unit tests for bytecode functions.
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
*
Bump copyright dates for 2025
2025-02-14 10:24:30 -05:00
* Copyright (C) 2013-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
Updating and cleaning up copyright notices.
2019-01-25 10:15:50 -05:00
* Copyright (C) 2009-2013 Sourcefire, Inc.
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
*
* Authors: Török Edvin
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#if HAVE_CONFIG_H
#include "clamav-config.h"
#endif
#include <stdio.h>
#include <stdlib.h>
#include <limits.h>
#include <string.h>
#include <check.h>
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
#include <fcntl.h>
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
#include <errno.h>
Use OpenSSL for hashing.
2014-02-08 00:31:12 -05:00
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
// libclamav
#include "clamav.h"
#include "others.h"
#include "bytecode.h"
#include "dconf.h"
#include "bytecode_priv.h"
#include "pe.h"
Allmatch-mode overhaul, part 1: append_virus Rework the append_virus mechanism to store evidence (strong indicators, pua indicators, and eventually weak indicators) in vectors. When appending a "virus", we will return CLEAN when in allmatch-mode, and simply add the indicator to the appropriate vector. Later we can check if there were any alerts to return a vector by summing the lengths of the strong and pua indicator vectors. This does away with storing the latest "virname" in the scan context. Instead, we can query for the last indicator in the evidence, giving priority to strong indicators. When heuristic-precendence is enabled, add PUA as Strong instead of as PotentiallyUnwanted. This way, they will be treated equally and reported in order in allmatch mode. Also document reason for disabling cache with metadata JSON enabled
2022-08-03 20:34:48 -07:00
#include "clamav_rust.h"
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
#include "checks.h"
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
Fix build error.
2010-11-04 22:13:35 +02:00
#ifdef CL_THREAD_SAFE
#include <pthread.h>
#endif
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
static void runtest(const char *file, uint64_t expected, int fail, int nojit,
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
const char *infile, struct cli_pe_hook_data *pedata,
struct cli_exe_section *sections, const char *expectedvirname,
int testmode)
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
{
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
fmap_t *map = NULL;
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
int rc;
Fix Windows text file EOL conversion issues On Windows, files open()'ed without the O_BINARY flag will have new-line LF (aka \n) converted to CRLF (aka \r\n) automatically when read from or written to. This is undesirable for all scan targets AND temp files because it affects pattern matching and with hashing. This commit converts a handful of instances throughout the codebase where it appears that O_BINARY was mistakenly omitted and could result in unexpected behavior on Windows. Git on Windows also converts LF -> CRLF for "text" files, for editing purposes. This is problematic for scan files and test files that should match verbatim. We can prevent this issue by marking .ref test files as "binary" in the .gitattributes file and by always opening scan files and temp files as binary. In this commit I've also removed the `ChangeLog merge=cl-merge` line that was once used to reduce ChangeLog merge conflicts by using the gnulib git-merge-changlog tool. This project now categorizes changes in the NEWS.md. For finer detail, git commit history is fully accessible on github.com.
2020-12-07 15:18:29 -08:00
int fd = open_testfile(file, O_RDONLY); /* CBC databases should be opened in text mode (not binary), or else CRLF line indexing with `fgets()` will fail. */
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
FILE *f;
struct cli_bc bc;
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
cli_ctx cctx;
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
struct cli_bc_ctx *ctx;
Draft of libclamav <-> jit communication.
2009-08-25 18:54:14 +03:00
struct cli_all_bc bcs;
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
uint64_t v;
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
struct cl_engine *engine;
Fix valgrind warnings.
2010-05-13 23:35:47 +03:00
int fdin = -1;
unit tests for new fmap scan API
2011-06-14 22:35:03 +03:00
char filestr[512];
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
const char *virname = NULL;
Restructured scan options flags from a single bitflag field to a structure containing multiple bitflag fields. This also required adding a new function to the bytecode API to get scan options a la carte, and modifying the existing function to hand back scan options in the old/deprecated uint32_t bitflag format. Re-generated bytecode iface header files. Updated libclamav documentation detailing new scan options structure. Renamed references to 'algorithmic' detection to 'heuristic' detection. Renaming references to 'properties' to 'collect metadata'. Renamed references to 'scan all' to 'scan all match'. Renamed a couple of 'Hueristic.*' signature names as 'Heuristics.*' signatures (plural) to match majority of other heuristics.
2018-07-20 22:28:48 -04:00
struct cl_scan_options options;
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
memset(&cctx, 0, sizeof(cctx));
Restructured scan options flags from a single bitflag field to a structure containing multiple bitflag fields. This also required adding a new function to the bytecode API to get scan options a la carte, and modifying the existing function to hand back scan options in the old/deprecated uint32_t bitflag format. Re-generated bytecode iface header files. Updated libclamav documentation detailing new scan options structure. Renamed references to 'algorithmic' detection to 'heuristic' detection. Renaming references to 'properties' to 'collect metadata'. Renamed references to 'scan all' to 'scan all match'. Renamed a couple of 'Hueristic.*' signature names as 'Heuristics.*' signatures (plural) to match majority of other heuristics.
2018-07-20 22:28:48 -04:00
memset(&options, 0, sizeof(struct cl_scan_options));
cctx.options = &options;
cctx.options->general |= CL_SCAN_GENERAL_ALLMATCHES;
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
cctx.engine = engine = cl_engine_new();
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!cctx.engine, "cannot create engine");
Fix interpreter.
2010-05-13 23:25:11 +03:00
rc = cl_engine_compile(engine);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!rc, "cannot compile engine");
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
cctx.dconf = cctx.engine->dconf;
cctx.recursion_stack_size = cctx.engine->max_recursion_level;
Remove max-allocation limits where not required The cli_max_malloc, cli_max_calloc, and cli_max_realloc functions provide a way to protect against allocating too much memory when the size of the allocation is derived from the untrusted input. Specifically, we worry about values in the file being scanned being manipulated to exhaust the RAM and crash the application. There is no need to check the limits if the size of the allocation is fixed, or if the size of the allocation is necessary for signature loading, or the general operation of the applications. E.g. checking the max-allocation limit for the size of a hash, or for the size of the scan recursion stack, is a complete waste of time. Although we significantly increased the max-allocation limit in a recent release, it is best not to check an allocation if the allocation will be safe. It would be a waste of time. I am also hopeful that if we can reduce the number allocations that require a limit-check to those that require it for the safe scan of a file, then eventually we can store the limit in the scan- context, and make it configurable.
2024-01-08 22:48:28 -05:00
cctx.recursion_stack = calloc(sizeof(recursion_level_t), cctx.recursion_stack_size);
ck_assert_msg(!!cctx.recursion_stack, "calloc() for recursion_stack failed");
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
// ctx was memset, so recursion_level starts at 0.
cctx.recursion_stack[cctx.recursion_level].fmap = NULL;
cctx.fmap = cctx.recursion_stack[cctx.recursion_level].fmap;
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(fd >= 0, "retmagic open failed");
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
f = fdopen(fd, "r");
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!f, "retmagic fdopen failed");
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
cl_debug();
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
if (!nojit) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
rc = cli_bytecode_init(&bcs);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(rc == CL_SUCCESS, "cli_bytecode_init failed");
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
} else {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
bcs.engine = NULL;
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
}
Draft of libclamav <-> jit communication.
2009-08-25 18:54:14 +03:00
bcs.all_bcs = &bc;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
bcs.count = 1;
Draft of libclamav <-> jit communication.
2009-08-25 18:54:14 +03:00
Add bytecode performance statistics
2012-12-05 15:48:52 -08:00
rc = cli_bytecode_load(&bc, f, NULL, 1, 0);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(rc == CL_SUCCESS, "cli_bytecode_load failed");
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
fclose(f);
Fix linker issues with global variable used in tests The `have_clamjit` global is used in the unit tests but doesn't appear to be exported when I was testing the external LLVM runtime support PR, resulting in an undefined symbol issue. Converting this to a function that returns 0 or 1 instead of a global variable resolved the issue.
2022-03-03 18:20:27 -08:00
if (testmode && have_clamjit())
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
engine->bytecode_mode = CL_BYTECODE_MODE_TEST;
Add engine param to bytecode, and remove dconf from _init.
2010-07-29 13:22:35 +03:00
freshclam is using private symbol that changed proto (bb #2187). Change name to prevent crash with 0.96.1 freshclam and 0.96.2 libclamav. You'll get a missing symbol error.
2010-08-11 13:39:49 +03:00
rc = cli_bytecode_prepare2(engine, &bcs, BYTECODE_ENGINE_MASK);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(rc == CL_SUCCESS, "cli_bytecode_prepare failed");
bytecode: fully switch to a stack based allocation, and operands with various bitwidths.
2009-07-23 17:33:11 +03:00
Tests: silence LGTM-com static analysis warning
2022-03-09 10:34:30 -08:00
if (have_clamjit() && !nojit && !testmode) {
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(bc.state == bc_jit, "preparing for JIT failed");
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
ctx = cli_bytecode_context_alloc();
Increase timeout of testcase itself.
2010-03-30 11:10:58 +03:00
ctx->bytecode_timeout = fail == CL_ETIMEOUT ? 10 : 10000;
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!ctx, "cli_bytecode_context_alloc failed");
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
ctx->ctx = &cctx;
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
if (infile) {
CMake: Add CTest support to match Autotools checks An ENABLE_TESTS CMake option is provided so that users can disable testing if they don't want it. Instructions for how to use this included in the INSTALL.cmake.md file. If you run `ctest`, each testcase will write out a log file to the <build>/unit_tests directory. As with Autotools' make check, the test files are from test/.split and unit_tests/.split files, but for CMake these are generated at build time instead of at test time. On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled libraries can be loaded when running tests. On Windows systems, CTest will identify and collect all library dependencies and assemble a temporarily install under the build/unit_tests directory so that the libraries can be loaded when running tests. The same feature is used on Windows when using CMake to install to collect all DLL dependencies so that users don't have to install them manually afterwards. Each of the CTest tests are run using a custom wrapper around Python's unittest framework, which is also responsible for finding and inserting valgrind into the valgrind tests on Posix systems. Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by default, if Valgrind can be found. There's no need to set VG=1. CTest's memcheck module is NOT supported, because we use Python to orchestrate our tests. Added a bunch of Windows compatibility changes to the unit tests. These were primarily changing / to PATHSEP and making adjustments to use Win32 C headers and ifdef out the POSIX ones which aren't available on Windows. Also disabled a bunch of tests on Win32 that don't work on Windows, notably the mmap ones and FD-passing (i.e. FILEDES) ones. Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate warnings on Windows where json.h is included after inttypes.h because json-c's inttypes replacement relies on it. This is a it of a hack and may be removed if json-c fixes their inttypes header stuff in the future. Add preprocessor definitions on Windows to disable MSVC warnings about CRT secure and nonstandard functions. While there may be a better solution, this is needed to be able to see other more serious warnings. Add missing file comment block and copyright statement for clamsubmit.c. Also change json-c/json.h include filename to json.h in clamsubmit.c. The directory name is not required. Changed the hash table data integer type from long, which is poorly defined, to size_t -- which is capable of storing a pointer. Fixed a bunch of casts regarding this variable to eliminate warnings. Fixed two bugs causing utf8 encoding unit tests to fail on Windows: - The in_size variable should be the number of bytes, not the character count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8 transcoding test to only transcode half the bytes. - It turns out that the MultiByteToWideChar() API can't transcode UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer and flip the bytes on each uint16_t. This but was causing the UTF16-BE to UTF8 tests to fail. I also split up the utf8 transcoding tests into separate tests so I could see all of the failures instead of just the first one. Added a flags parameter to the unit test function to open testfiles because it turns out that on Windows if a file contains the \r\n it will replace it with just \n if you opened the file as a text file instead of as binary. However, if we open the CBC files as binary, then a bunch of bytecode tests fail. So I've changed the tests to open the CBC files in the bytecode tests as text files and open all other files as binary. Ported the feature tests from shell scripts to Python using a modified version of our QA test-framework, which is largely compatible and will allow us to migrate some QA tests into this repo. I'd like to add GitHub Actions pipelines in the future so that all public PR's get some testing before anyone has to manually review them. The clamd --log option was missing from the help string, though it definitely works. I've added it in this commit. It appears that clamd.c was never clang-format'd, so this commit also reformats clamd.c. Some of the check_clamd tests expected the path returned by clamd to match character for character with original path sent to clamd. However, as we now evaluate real paths before a scan, the path returned by clamd isn't going to match the relative (and possibly symlink-ridden) path passed to clamdscan. I fixed this test by changing the test to search for the basename: <signature> FOUND within the response instead of matching the exact path. Autotools: Link check_clamd with libclamav so we can use our utility functions in check_clamd.c.
2020-08-25 23:14:23 -07:00
snprintf(filestr, sizeof(filestr), OBJDIR PATHSEP "%s", infile);
Fix Windows text file EOL conversion issues On Windows, files open()'ed without the O_BINARY flag will have new-line LF (aka \n) converted to CRLF (aka \r\n) automatically when read from or written to. This is undesirable for all scan targets AND temp files because it affects pattern matching and with hashing. This commit converts a handful of instances throughout the codebase where it appears that O_BINARY was mistakenly omitted and could result in unexpected behavior on Windows. Git on Windows also converts LF -> CRLF for "text" files, for editing purposes. This is problematic for scan files and test files that should match verbatim. We can prevent this issue by marking .ref test files as "binary" in the .gitattributes file and by always opening scan files and temp files as binary. In this commit I've also removed the `ChangeLog merge=cl-merge` line that was once used to reduce ChangeLog merge conflicts by using the gnulib git-merge-changlog tool. This project now categorizes changes in the NEWS.md. For finer detail, git commit history is fully accessible on github.com.
2020-12-07 15:18:29 -08:00
fdin = open(filestr, O_RDONLY | O_BINARY);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (fdin < 0 && errno == ENOENT)
Fix Windows text file EOL conversion issues On Windows, files open()'ed without the O_BINARY flag will have new-line LF (aka \n) converted to CRLF (aka \r\n) automatically when read from or written to. This is undesirable for all scan targets AND temp files because it affects pattern matching and with hashing. This commit converts a handful of instances throughout the codebase where it appears that O_BINARY was mistakenly omitted and could result in unexpected behavior on Windows. Git on Windows also converts LF -> CRLF for "text" files, for editing purposes. This is problematic for scan files and test files that should match verbatim. We can prevent this issue by marking .ref test files as "binary" in the .gitattributes file and by always opening scan files and temp files as binary. In this commit I've also removed the `ChangeLog merge=cl-merge` line that was once used to reduce ChangeLog merge conflicts by using the gnulib git-merge-changlog tool. This project now categorizes changes in the NEWS.md. For finer detail, git commit history is fully accessible on github.com.
2020-12-07 15:18:29 -08:00
fdin = open_testfile(infile, O_RDONLY | O_BINARY);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(fdin >= 0, "failed to open infile");
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
map = fmap_new(fdin, 0, 0, filestr, NULL);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!map, "unable to fmap infile");
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
if (pedata)
ctx->hooks.pedata = pedata;
ctx->sections = sections;
cli_bytecode_context_setfile(ctx, map);
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
}
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
cli_bytecode_context_setfuncid(ctx, &bc, 0);
really execute the JITed code.
2009-08-27 20:41:29 +03:00
rc = cli_bytecode_run(&bcs, &bc, ctx);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(rc == fail, "cli_bytecode_run failed, expected: %u, have: %u\n",
Clang-format touchup
2020-07-24 08:32:47 -07:00
fail, rc);
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
Handle runtime errors with setjmp/longjmp, using a thread-local jmpbuf to make it threadsafe.
2009-08-28 13:18:17 +03:00
if (rc == CL_SUCCESS) {
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
v = cli_bytecode_context_getresult_int(ctx);
DB read logic cleanup, fix some warnings The logic for parsing a logical subsignature isn't clearly identified and has been, perhaps mistakenly or out of convenience, used to when parsing NDB signatures in addition to LDB subsignatures. What this means is that you can technically use a PCRE subsignature in an NDB file and clam won't complain about it. It won't work however, because a PCRE subsignature requires another matching subsignature to trigger it, but it will parse. The same is likely true for byte-compare subsignatures. This commit restructures that logic a bit so subsignature parsing has its own function and is more organized. I also renamed the functions a little bit and added lots of comments. I fixed a few minor warnings relating to format string characters. The change in str.c:cli_ldbtokenize is to prevent a buffer under-read if you were to use the function on the start of a buffer, as is now down in this commit.
2022-02-12 13:34:25 -08:00
ck_assert_msg(v == expected, "Invalid return value from bytecode run, expected: " STDx64 ", have: " STDx64 "\n",
Clang-format touchup
2020-07-24 08:32:47 -07:00
expected, v);
Handle runtime errors with setjmp/longjmp, using a thread-local jmpbuf to make it threadsafe.
2009-08-28 13:18:17 +03:00
}
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
if (infile && expectedvirname) {
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(ctx->virname &&
Clang-format touchup
2020-07-24 08:32:47 -07:00
!strcmp(ctx->virname, expectedvirname),
"Invalid virname, expected: %s\n", expectedvirname);
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
}
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
cli_bytecode_context_destroy(ctx);
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
if (map)
Record unique object-id for each layer scanned Every time we push a new map onto the scanning recursion context, give it a unique object id number, which counts from zero. Moved the location where we add metadata for each file from the "cli_magic_scan" function over to the "recursion stack push" function. Include a "path" as a parameter for creating a new fmap, and rename some related variables and functions to be more intuitive. CLAM-2796 See also: CLAM-2485, CLAM-2626
2025-06-08 01:12:33 -04:00
fmap_free(map);
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
cli_bytecode_destroy(&bc);
Draft of libclamav <-> jit communication.
2009-08-25 18:54:14 +03:00
cli_bytecode_done(&bcs);
Record scan matches (evidence) at each recursion layer Move recording of evidence (aka Strong, PUA, and Weak indicators) to be done in each layer of a scan, and passed up to the parent layer with the top level only connecting the results at the very end of the scan. This is needed to provide access the last alert for a given layer when we upgrade the scan callbacks. Note that when adding evidence from a child layer that is a normalized layer, we do not want to increase the depth. It should appear as though the match occured on the parent layer. This is for two reasons: 1. We don't run the scan callbacks on normalized layers. 2. Future matches on Weak Indicators should be able to treat normalized layer matches the same as original file matches. Keep reading for more about Weak Indicators. Recording scan matches at each recursion layer is also needed to support Weak Indicators, a feature where an alerting signature (aka Strong Indicator) may require the the match of a non-alerting signature (aka Weak Indicator) on the same layer or on child layers in order to alert. Support for Weak indicators was blocked by not keeping track of where indicators were found. So this commit also enables support for recording Weak indicators. Like PUA, Weak indicators are treated differently based on the signature prefix. That is, any signatures starting with "Weak." won't cause an alert on its own. The next step to completing Weak Indicator support will be adding a logical subsignature feature to depend on a weak indicator match. CLAM-2626 CLAM-2485
2025-06-09 01:33:26 -04:00
if (cctx.recursion_stack[cctx.recursion_level].evidence) {
evidence_free(cctx.recursion_stack[cctx.recursion_level].evidence);
}
libclamav: Fix scan recursion tracking Scan recursion is the process of identifying files embedded in other files and then scanning them, recursively. Internally this process is more complex than it may sound because a file may have multiple layers of types before finding a new "file". At present we treat the recursion count in the scanning context as an index into both our fmap list AND our container list. These two lists are conceptually a part of the same thing and should be unified. But what's concerning is that the "recursion level" isn't actually incremented or decremented at the same time that we add a layer to the fmap or container lists but instead is more touchy-feely, increasing when we find a new "file". To account for this shadiness, the size of the fmap and container lists has always been a little longer than our "max scan recursion" limit so we don't accidentally overflow the fmap or container arrays (!). I've implemented a single recursion-stack as an array, similar to before, which includes a pointer to each fmap at each layer, along with the size and type. Push and pop functions add and remove layers whenever a new fmap is added. A boolean argument when pushing indicates if the new layer represents a new buffer or new file (descriptor). A new buffer will reset the "nested fmap level" (described below). This commit also provides a solution for an issue where we detect embedded files more than once during scan recursion. For illustration, imagine a tarball named foo.tar.gz with this structure: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | baz.exe | PE | 0 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | │   └── hello.txt | ASCII | 2 | 0 | | └── sfx.7z | 7Z | 1 | 1 | |    └── world.txt | ASCII | 2 | 0 | (A) If we scan for embedded files at any layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | ├── foo.tar | TAR | 1 | 0 | | │ ├── bar.zip | ZIP | 2 | 1 | | │ │   └── hola.txt | ASCII | 3 | 0 | | │ ├── baz.exe | PE | 2 | 1 | | │ │ ├── sfx.zip | ZIP | 3 | 1 | | │ │ │   └── hello.txt | ASCII | 4 | 0 | | │ │ └── sfx.7z | 7Z | 3 | 1 | | │ │    └── world.txt | ASCII | 4 | 0 | | │ ├── sfx.zip | ZIP | 2 | 1 | | │ │   └── hello.txt | ASCII | 3 | 0 | | │ └── sfx.7z | 7Z | 2 | 1 | | │   └── world.txt | ASCII | 3 | 0 | | ├── sfx.zip | ZIP | 1 | 1 | | └── sfx.7z | 7Z | 1 | 1 | (A) is bad because it scans content more than once. Note that for the GZ layer, it may detect the ZIP and 7Z if the signature hits on the compressed data, which it might, though extracting the ZIP and 7Z will likely fail. The reason the above doesn't happen now is that we restrict embedded type scans for a bunch of archive formats to include GZ and TAR. (B) If we scan for embedded files at the foo.tar layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | ├── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 2 | 1 | | │   └── hello.txt | ASCII | 3 | 0 | | └── sfx.7z | 7Z | 2 | 1 | |    └── world.txt | ASCII | 3 | 0 | (B) is almost right. But we can achieve it easily enough only scanning for embedded content in the current fmap when the "nested fmap level" is 0. The upside is that it should safely detect all embedded content, even if it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe. The biggest risk I can think of affects ZIPs. SFXZIP detection is identical to ZIP detection, which is why we don't allow SFXZIP to be detected if insize of a ZIP. If we only allow embedded type scanning at fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP if the bar.exe was not compressed in foo.zip and if non-compressed files extracted from ZIPs aren't extracted as new buffers: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.zip | ZIP | 0 | 0 | | └── bar.exe | PE | 1 | 1 | | └── sfx.zip | ZIP | 2 | 2 | Provided that we ensure all files extracted from zips are scanned in new buffers, option (B) should be safe. (C) If we scan for embedded files at the baz.exe layer, we may detect: | description | type | rec level | nested fmap level | | ------------------------- | ----- | --------- | ----------------- | | foo.tar.gz | GZ | 0 | 0 | | └── foo.tar | TAR | 1 | 0 | | ├── bar.zip | ZIP | 2 | 1 | | │   └── hola.txt | ASCII | 3 | 0 | | └── baz.exe | PE | 2 | 1 | | ├── sfx.zip | ZIP | 3 | 1 | | │   └── hello.txt | ASCII | 4 | 0 | | └── sfx.7z | 7Z | 3 | 1 | |    └── world.txt | ASCII | 4 | 0 | (C) is right. But it's harder to achieve. For this example we can get it by restricting 7ZSFX and ZIPSFX detection only when scanning an executable. But that may mean losing detection of archives embedded elsewhere. And we'd have to identify allowable container types for each possible embedded type, which would be very difficult. So this commit aims to solve the issue the (B)-way. Note that in all situations, we still have to scan with file typing enabled to determine if we need to reassign the current file type, such as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2- compressed. Detection of DMG and a handful of other types rely on finding data partway through or near the ned of a file before reassigning the entire file as the new type. Other fixes and considerations in this commit: - The utf16 HTML parser has weak error handling, particularly with respect to creating a nested fmap for scanning the ascii decoded file. This commit cleans up the error handling and wraps the nested scan with the recursion-stack push()/pop() for correct recursion tracking. Before this commit, each container layer had a flag to indicate if the container layer is valid. We need something similar so that the cli_recursion_stack_get_*() functions ignore normalized layers. Details... Imagine an LDB signature for HTML content that specifies a ZIP container. If the signature actually alerts on the normalized HTML and you don't ignore normalized layers for the container check, it will appear as though the alert is in an HTML container rather than a ZIP container. This commit accomplishes this with a boolean you set in the scan context before scanning a new layer. Then when the new fmap is created, it will use that flag to set similar flag for the layer. The context flag is reset those that anything after this doesn't have that flag. The flag allows the new recursion_stack_get() function to ignore normalized layers when iterating the stack to return a layer at a requested index, negative or positive. Scanning normalized extracted/normalized javascript and VBA should also use the 'layer is normalized' flag. - This commit also fixes Heuristic.Broken.Executable alert for ELF files to make sure that: A) these only alert if cli_append_virus() returns CL_VIRUS (aka it respects the FP check). B) all broken-executable alerts for ELF only happen if the SCAN_HEURISTIC_BROKEN option is enabled. - This commit also cleans up the error handling in cli_magic_scan_dir(). This was needed so we could correctly apply the layer-is-normalized-flag to all VBA macros extracted to a directory when scanning the directory. - Also fix an issue where exceeding scan maximums wouldn't cause embedded file detection scans to abort. Granted we don't actually want to abort if max filesize or max recursion depth are exceeded... only if max scansize, max files, and max scantime are exceeded. Add 'abort_scan' flag to scan context, to protect against depending on correct error propagation for fatal conditions. Instead, setting this flag in the scan context should guarantee that a fatal condition deep in scan recursion isn't lost which result in more stuff being scanned instead of aborting. This shouldn't be necessary, but some status codes like CL_ETIMEOUT never used to be fatal and it's easier to do this than to verify every parser only returns CL_ETIMEOUT and other "fatal status codes" in fatal conditions. - Remove duplicate is_tar() prototype from filestypes.c and include is_tar.h instead. - Presently we create the fmap hash when creating the fmap. This wastes a bit of CPU if the hash is never needed. Now that we're creating fmap's for all embedded files discovered with file type recognition scans, this is a much more frequent occurence and really slows things down. This commit fixes the issue by only creating fmap hashes as needed. This should not only resolve the perfomance impact of creating fmap's for all embedded files, but also should improve performance in general. - Add allmatch check to the zip parser after the central-header meta match. That way we don't multiple alerts with the same match except in allmatch mode. Clean up error handling in the zip parser a tiny bit. - Fixes to ensure that the scan limits such as scansize, filesize, recursion depth, # of embedded files, and scantime are always reported if AlertExceedsMax (--alert-exceeds-max) is enabled. - Fixed an issue where non-fatal alerts for exceeding scan maximums may mask signature matches later on. I changed it so these alerts use the "possibly unwanted" alert-type and thus only alert if no other alerts were found or if all-match or heuristic-precedence are enabled. - Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata when the --gen-json feature is enabled. These will show up once under "ParseErrors" the first time a limit is exceeded. In the present implementation, only one limits-exceeded events will be added, so as to prevent a malicious or malformed sample from filling the JSON buffer with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
free(cctx.recursion_stack);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
cl_engine_free(engine);
Fix valgrind warnings.
2010-05-13 23:35:47 +03:00
if (fdin >= 0)
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
close(fdin);
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_retmagic_jit)
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
{
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "retmagic.cbc", 0x1234f00d, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 0);
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "retmagic.cbc", 0x1234f00d, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 1);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_retmagic_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "retmagic.cbc", 0x1234f00d, CL_SUCCESS, 1, NULL, NULL, NULL, NULL, 0);
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_arith_jit)
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
{
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "arith.cbc", 0xd5555555, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_arith_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "arith.cbc", 0xd5555555, CL_SUCCESS, 1, NULL, NULL, NULL, NULL, 0);
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
}
END_TEST
Implement API calls from bytecode.
2009-08-20 16:23:43 +03:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls_jit)
Implement API calls from bytecode.
2009-08-20 16:23:43 +03:00
{
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls.cbc", 0xf00d, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls.cbc", 0xf00d, CL_SUCCESS, 1, NULL, NULL, NULL, NULL, 0);
Implement API calls from bytecode.
2009-08-20 16:23:43 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls2_jit)
Apicall0 suppport for JIT.
2009-09-02 18:53:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls2.cbc", 0xf00d, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls2_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls2.cbc", 0xf00d, CL_SUCCESS, 1, NULL, NULL, NULL, NULL, 0);
Apicall0 suppport for JIT.
2009-09-02 18:53:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_div0_jit)
Handle runtime errors with setjmp/longjmp, using a thread-local jmpbuf to make it threadsafe.
2009-08-28 13:18:17 +03:00
{
Various bytecode JIT fixes, teach clamconf about JIT, and make sure make check runs the JIT!
2009-08-28 20:07:25 +03:00
cl_init(CL_INIT_DEFAULT);
Handle runtime errors with setjmp/longjmp, using a thread-local jmpbuf to make it threadsafe.
2009-08-28 13:18:17 +03:00
/* must not crash on div#0 but catch it */
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "div0.cbc", 0, CL_EBYTECODE, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_div0_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "div0.cbc", 0, CL_EBYTECODE, 1, NULL, NULL, NULL, NULL, 0);
Handle runtime errors with setjmp/longjmp, using a thread-local jmpbuf to make it threadsafe.
2009-08-28 13:18:17 +03:00
}
END_TEST
Implement API calls from bytecode.
2009-08-20 16:23:43 +03:00
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_lsig_jit)
Initial support for loading .cbc files from DB.
2009-09-21 19:24:16 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "lsig.cbc", 0, 0, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_lsig_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "lsig.cbc", 0, 0, 1, NULL, NULL, NULL, NULL, 0);
Initial support for loading .cbc files from DB.
2009-09-21 19:24:16 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_inf_jit)
Introduce BytecodeTimeout.
2010-03-22 17:16:07 +02:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "inf.cbc", 0, CL_ETIMEOUT, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_inf_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "inf.cbc", 0, CL_ETIMEOUT, 1, NULL, NULL, NULL, NULL, 0);
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_matchwithread_jit)
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
{
struct cli_exe_section sect;
struct cli_pe_hook_data pedata;
cl_init(CL_INIT_DEFAULT);
memset(&pedata, 0, sizeof(pedata));
pedata.ep = 64;
Fix matchwithread.cbc ImageBase is little-endian, need to use conversion function to access it.
2010-04-02 13:07:23 +03:00
cli_writeint32(&pedata.opt32.ImageBase, 0x400000);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
pedata.hdr_size = 0x400;
Add matchwithread.cbc to unit tests.
2010-03-24 12:46:34 +02:00
pedata.nsections = 1;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
sect.rva = 4096;
sect.vsz = 4096;
sect.raw = 0;
sect.rsz = 512;
sect.urva = 4096;
sect.uvsz = 4096;
sect.uraw = 1;
sect.ursz = 512;
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "matchwithread.cbc", 0, 0, 0,
"input" PATHSEP "clamav_hdb_scanfiles" PATHSEP "clam.exe",
&pedata, &sect, "ClamAV-Test-File-detected-via-bytecode", 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_matchwithread_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
struct cli_exe_section sect;
struct cli_pe_hook_data pedata;
cl_init(CL_INIT_DEFAULT);
memset(&pedata, 0, sizeof(pedata));
pedata.ep = 64;
Fix matchwithread.cbc ImageBase is little-endian, need to use conversion function to access it.
2010-04-02 13:07:23 +03:00
cli_writeint32(&pedata.opt32.ImageBase, 0x400000);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
pedata.hdr_size = 0x400;
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
pedata.nsections = 1;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
sect.rva = 4096;
sect.vsz = 4096;
sect.raw = 0;
sect.rsz = 512;
sect.urva = 4096;
sect.uvsz = 4096;
sect.uraw = 1;
sect.ursz = 512;
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "matchwithread.cbc", 0, 0, 1,
"input" PATHSEP "clamav_hdb_scanfiles" PATHSEP "clam.exe",
&pedata, &sect, "ClamAV-Test-File-detected-via-bytecode", 0);
Introduce BytecodeTimeout.
2010-03-22 17:16:07 +02:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_pdf_jit)
Leak testcase.
2010-03-24 14:14:33 +02:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "pdf.cbc", 0, 0, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_pdf_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "pdf.cbc", 0, 0, 1, NULL, NULL, NULL, NULL, 0);
Leak testcase.
2010-03-24 14:14:33 +02:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_bswap_jit)
Fix bswap.
2010-03-24 15:27:15 +02:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "bswap.cbc", 0xbeef, 0, 0, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_bswap_int)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "bswap.cbc", 0xbeef, 0, 1, NULL, NULL, NULL, NULL, 0);
Fix bswap.
2010-03-24 15:27:15 +02:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_inflate_jit)
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "inflate.cbc", 0xbeef, 0, 1, NULL, NULL, NULL, NULL, 0);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_inflate_int)
Fix more leaks.
2010-03-24 17:07:14 +02:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "inflate.cbc", 0xbeef, 0, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_api_extract_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "api_extract_7.cbc", 0xf00d, 0, 0,
"input" PATHSEP "bytecode_scanfiles" PATHSEP "apitestfile", NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_api_files_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "api_files_7.cbc", 0xf00d, 0, 0,
"input" PATHSEP "bytecode_scanfiles" PATHSEP "apitestfile", NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls2_7_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls2_7.cbc", 0xf00d, 0, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls_7_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls_7.cbc", 0xf00d, 0, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_arith_7_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "arith_7.cbc", 0xd55555dd, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_debug_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "debug_7.cbc", 0xf00d, 0, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_inf_7_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "inf_7.cbc", 0, CL_ETIMEOUT, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_lsig_7_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "lsig_7.cbc", 0, 0, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_retmagic_7_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "retmagic_7.cbc", 0x1234f00d, CL_SUCCESS, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_testadt_jit)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "testadt_7.cbc", 0xf00d, 0, 0, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_api_extract_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "api_extract_7.cbc", 0xf00d, 0, 1,
"input" PATHSEP "bytecode_scanfiles" PATHSEP "apitestfile", NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_api_files_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "api_files_7.cbc", 0xf00d, 0, 1,
"input" PATHSEP "bytecode_scanfiles" PATHSEP "apitestfile", NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls2_7_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls2_7.cbc", 0xf00d, 0, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_apicalls_7_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "apicalls_7.cbc", 0xf00d, 0, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_arith_7_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "arith_7.cbc", 0xd55555dd, CL_SUCCESS, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_debug_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "debug_7.cbc", 0xf00d, 0, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_inf_7_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "inf_7.cbc", 0, CL_ETIMEOUT, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_lsig_7_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "lsig_7.cbc", 0, 0, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_retmagic_7_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "retmagic_7.cbc", 0x1234f00d, CL_SUCCESS, 1, NULL, NULL, NULL, NULL, 0);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_testadt_int)
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
{
cl_init(CL_INIT_DEFAULT);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runtest("input" PATHSEP "bytecode_sigs" PATHSEP "testadt_7.cbc", 0xf00d, 0, 1, NULL, NULL, NULL, NULL, 0);
Fix more leaks.
2010-03-24 17:07:14 +02:00
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
static void runload(const char *dbname, struct cl_engine *engine, unsigned signoexp)
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
{
char *str;
unsigned signo = 0;
int rc;
Tests: remove dead code Remove check for 'srcdir' and 'unrar_disabled' variables. These were only used by legacy Automake tooling. Resolves: https://github.com/Cisco-Talos/clamav/issues/1447
2025-02-03 13:25:31 -05:00
str = malloc(strlen(SRCDIR) + 1 + strlen(dbname) + 1);
Remove max-allocation limits where not required The cli_max_malloc, cli_max_calloc, and cli_max_realloc functions provide a way to protect against allocating too much memory when the size of the allocation is derived from the untrusted input. Specifically, we worry about values in the file being scanned being manipulated to exhaust the RAM and crash the application. There is no need to check the limits if the size of the allocation is fixed, or if the size of the allocation is necessary for signature loading, or the general operation of the applications. E.g. checking the max-allocation limit for the size of a hash, or for the size of the scan recursion stack, is a complete waste of time. Although we significantly increased the max-allocation limit in a recent release, it is best not to check an allocation if the allocation will be safe. It would be a waste of time. I am also hopeful that if we can reduce the number allocations that require a limit-check to those that require it for the safe scan of a file, then eventually we can store the limit in the scan- context, and make it configurable.
2024-01-08 22:48:28 -05:00
ck_assert_msg(!!str, "malloc");
Tests: remove dead code Remove check for 'srcdir' and 'unrar_disabled' variables. These were only used by legacy Automake tooling. Resolves: https://github.com/Cisco-Talos/clamav/issues/1447
2025-02-03 13:25:31 -05:00
sprintf(str, "%s" PATHSEP "%s", SRCDIR, dbname);
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
rc = cl_load(str, engine, &signo, CL_DB_STDOPT);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(rc == CL_SUCCESS, "failed to load %s: %s\n",
CMake: Add CTest support to match Autotools checks An ENABLE_TESTS CMake option is provided so that users can disable testing if they don't want it. Instructions for how to use this included in the INSTALL.cmake.md file. If you run `ctest`, each testcase will write out a log file to the <build>/unit_tests directory. As with Autotools' make check, the test files are from test/.split and unit_tests/.split files, but for CMake these are generated at build time instead of at test time. On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled libraries can be loaded when running tests. On Windows systems, CTest will identify and collect all library dependencies and assemble a temporarily install under the build/unit_tests directory so that the libraries can be loaded when running tests. The same feature is used on Windows when using CMake to install to collect all DLL dependencies so that users don't have to install them manually afterwards. Each of the CTest tests are run using a custom wrapper around Python's unittest framework, which is also responsible for finding and inserting valgrind into the valgrind tests on Posix systems. Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by default, if Valgrind can be found. There's no need to set VG=1. CTest's memcheck module is NOT supported, because we use Python to orchestrate our tests. Added a bunch of Windows compatibility changes to the unit tests. These were primarily changing / to PATHSEP and making adjustments to use Win32 C headers and ifdef out the POSIX ones which aren't available on Windows. Also disabled a bunch of tests on Win32 that don't work on Windows, notably the mmap ones and FD-passing (i.e. FILEDES) ones. Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate warnings on Windows where json.h is included after inttypes.h because json-c's inttypes replacement relies on it. This is a it of a hack and may be removed if json-c fixes their inttypes header stuff in the future. Add preprocessor definitions on Windows to disable MSVC warnings about CRT secure and nonstandard functions. While there may be a better solution, this is needed to be able to see other more serious warnings. Add missing file comment block and copyright statement for clamsubmit.c. Also change json-c/json.h include filename to json.h in clamsubmit.c. The directory name is not required. Changed the hash table data integer type from long, which is poorly defined, to size_t -- which is capable of storing a pointer. Fixed a bunch of casts regarding this variable to eliminate warnings. Fixed two bugs causing utf8 encoding unit tests to fail on Windows: - The in_size variable should be the number of bytes, not the character count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8 transcoding test to only transcode half the bytes. - It turns out that the MultiByteToWideChar() API can't transcode UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer and flip the bytes on each uint16_t. This but was causing the UTF16-BE to UTF8 tests to fail. I also split up the utf8 transcoding tests into separate tests so I could see all of the failures instead of just the first one. Added a flags parameter to the unit test function to open testfiles because it turns out that on Windows if a file contains the \r\n it will replace it with just \n if you opened the file as a text file instead of as binary. However, if we open the CBC files as binary, then a bunch of bytecode tests fail. So I've changed the tests to open the CBC files in the bytecode tests as text files and open all other files as binary. Ported the feature tests from shell scripts to Python using a modified version of our QA test-framework, which is largely compatible and will allow us to migrate some QA tests into this repo. I'd like to add GitHub Actions pipelines in the future so that all public PR's get some testing before anyone has to manually review them. The clamd --log option was missing from the help string, though it definitely works. I've added it in this commit. It appears that clamd.c was never clang-format'd, so this commit also reformats clamd.c. Some of the check_clamd tests expected the path returned by clamd to match character for character with original path sent to clamd. However, as we now evaluate real paths before a scan, the path returned by clamd isn't going to match the relative (and possibly symlink-ridden) path passed to clamdscan. I fixed this test by changing the test to search for the basename: <signature> FOUND within the response instead of matching the exact path. Autotools: Link check_clamd with libclamav so we can use our utility functions in check_clamd.c.
2020-08-25 23:14:23 -07:00
str, cl_strerror(rc));
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(signo == signoexp, "different number of signatures loaded, expected %u, got %u\n",
Clang-format touchup
2020-07-24 08:32:47 -07:00
signoexp, signo);
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
free(str);
rc = cl_engine_compile(engine);
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(rc == CL_SUCCESS, "failed to load %s: %s\n",
CMake: Add CTest support to match Autotools checks An ENABLE_TESTS CMake option is provided so that users can disable testing if they don't want it. Instructions for how to use this included in the INSTALL.cmake.md file. If you run `ctest`, each testcase will write out a log file to the <build>/unit_tests directory. As with Autotools' make check, the test files are from test/.split and unit_tests/.split files, but for CMake these are generated at build time instead of at test time. On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled libraries can be loaded when running tests. On Windows systems, CTest will identify and collect all library dependencies and assemble a temporarily install under the build/unit_tests directory so that the libraries can be loaded when running tests. The same feature is used on Windows when using CMake to install to collect all DLL dependencies so that users don't have to install them manually afterwards. Each of the CTest tests are run using a custom wrapper around Python's unittest framework, which is also responsible for finding and inserting valgrind into the valgrind tests on Posix systems. Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by default, if Valgrind can be found. There's no need to set VG=1. CTest's memcheck module is NOT supported, because we use Python to orchestrate our tests. Added a bunch of Windows compatibility changes to the unit tests. These were primarily changing / to PATHSEP and making adjustments to use Win32 C headers and ifdef out the POSIX ones which aren't available on Windows. Also disabled a bunch of tests on Win32 that don't work on Windows, notably the mmap ones and FD-passing (i.e. FILEDES) ones. Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate warnings on Windows where json.h is included after inttypes.h because json-c's inttypes replacement relies on it. This is a it of a hack and may be removed if json-c fixes their inttypes header stuff in the future. Add preprocessor definitions on Windows to disable MSVC warnings about CRT secure and nonstandard functions. While there may be a better solution, this is needed to be able to see other more serious warnings. Add missing file comment block and copyright statement for clamsubmit.c. Also change json-c/json.h include filename to json.h in clamsubmit.c. The directory name is not required. Changed the hash table data integer type from long, which is poorly defined, to size_t -- which is capable of storing a pointer. Fixed a bunch of casts regarding this variable to eliminate warnings. Fixed two bugs causing utf8 encoding unit tests to fail on Windows: - The in_size variable should be the number of bytes, not the character count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8 transcoding test to only transcode half the bytes. - It turns out that the MultiByteToWideChar() API can't transcode UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer and flip the bytes on each uint16_t. This but was causing the UTF16-BE to UTF8 tests to fail. I also split up the utf8 transcoding tests into separate tests so I could see all of the failures instead of just the first one. Added a flags parameter to the unit test function to open testfiles because it turns out that on Windows if a file contains the \r\n it will replace it with just \n if you opened the file as a text file instead of as binary. However, if we open the CBC files as binary, then a bunch of bytecode tests fail. So I've changed the tests to open the CBC files in the bytecode tests as text files and open all other files as binary. Ported the feature tests from shell scripts to Python using a modified version of our QA test-framework, which is largely compatible and will allow us to migrate some QA tests into this repo. I'd like to add GitHub Actions pipelines in the future so that all public PR's get some testing before anyone has to manually review them. The clamd --log option was missing from the help string, though it definitely works. I've added it in this commit. It appears that clamd.c was never clang-format'd, so this commit also reformats clamd.c. Some of the check_clamd tests expected the path returned by clamd to match character for character with original path sent to clamd. However, as we now evaluate real paths before a scan, the path returned by clamd isn't going to match the relative (and possibly symlink-ridden) path passed to clamdscan. I fixed this test by changing the test to search for the basename: <signature> FOUND within the response instead of matching the exact path. Autotools: Link check_clamd with libclamav so we can use our utility functions in check_clamd.c.
2020-08-25 23:14:23 -07:00
str, cl_strerror(rc));
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_load_bytecode_jit)
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
{
struct cl_engine *engine;
cl_init(CL_INIT_DEFAULT);
engine = cl_engine_new();
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!engine, "failed to create engine\n");
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runload("input" PATHSEP "bytecode_sigs" PATHSEP "bytecode.cvd", engine, 5);
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
cl_engine_free(engine);
}
END_TEST
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_load_bytecode_int)
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
{
struct cl_engine *engine;
cl_init(CL_INIT_DEFAULT);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
engine = cl_engine_new();
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
engine->dconf->bytecode = BYTECODE_INTERPRETER;
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!engine, "failed to create engine\n");
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runload("input" PATHSEP "bytecode_sigs" PATHSEP "bytecode.cvd", engine, 5);
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
cl_engine_free(engine);
}
END_TEST
Fix build on Etch (bb #2399).
2011-01-20 10:02:06 +02:00
#if defined(CL_THREAD_SAFE) && defined(C_LINUX) && ((__GLIBC__ << 16) + __GLIBC_MINOR__ >= (2 << 16) + 4)
Restrict usage of pthread_barrier even more. Don't want to add a configure test for it now...
2010-11-23 13:31:47 +02:00
#define DO_BARRIER
#endif
#ifdef DO_BARRIER
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
static pthread_barrier_t barrier;
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
static void *thread(void *arg)
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
{
struct cl_engine *engine;
engine = cl_engine_new();
Check test support for check 0.13 Tests in libcheck 0.13 must have {} between START_TEST and END_TEST else it will not compile. Also replaced all deprecated "fail_" macros with "ck_" macros. E.g. fail_unless() becomes ck_assert_msg() The checks_common.h header file provided a couple of macros to support versions older than 0.9.3. As these older versions are no longer relevant, I've removed those compatibility macros entirely.
2020-01-15 08:14:23 -08:00
ck_assert_msg(!!engine, "failed to create engine\n");
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
/* run all cl_load at once, to maximize chance of a crash
* in case of a race condition */
pthread_barrier_wait(&barrier);
XOR test files; clean up tests directory The split test files are flagged by some AV's because they look like broken executables. Instead of splitting the test files to prevent detections, we should encrypt them. This commit replaces the "reassemble testfiles" script with a basic "XOR testfiles" script that can be used to encrypt or decrypt test files. This commit also of course then replaces all the split files with xor'ed files. The test and unit_tests directories were a bit of a mess, so I reorganized them all into unit_tests with all of the test files placed under "unit_tests/input" using subdirectories for different types of files.
2021-07-16 13:43:28 -07:00
runload("input" PATHSEP "bytecode_sigs" PATHSEP "bytecode.cvd", engine, 5);
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
cl_engine_free(engine);
return NULL;
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
START_TEST(test_parallel_load)
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
{
#define N 5
pthread_t threads[N];
unsigned i;
cl_init(CL_INIT_DEFAULT);
pthread_barrier_init(&barrier, NULL, N);
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
for (i = 0; i < N; i++) {
pthread_create(&threads[i], NULL, thread, NULL);
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
}
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
for (i = 0; i < N; i++) {
pthread_join(threads[i], NULL);
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
}
/* DB load used to crash due to 'static' variable in cache.c,
* and also due to something wrong in LLVM 2.7.
* Enabled the mutex around codegen in bytecode2llvm.cpp, and this test is
* here to make sure it doesn't crash */
}
END_TEST
#endif
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
Suite *test_bytecode_suite(void)
{
clang-format'd using new .clang-format rules.
2018-12-03 12:40:13 -05:00
Suite *s = suite_create("bytecode");
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
TCase *tc_cli_arith = tcase_create("arithmetic");
suite_add_tcase(s, tc_cli_arith);
Increase timeout of testcase itself.
2010-03-30 11:10:58 +03:00
tcase_set_timeout(tc_cli_arith, 20);
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
tcase_add_test(tc_cli_arith, test_retmagic_jit);
tcase_add_test(tc_cli_arith, test_arith_jit);
tcase_add_test(tc_cli_arith, test_apicalls_jit);
tcase_add_test(tc_cli_arith, test_apicalls2_jit);
tcase_add_test(tc_cli_arith, test_div0_jit);
tcase_add_test(tc_cli_arith, test_lsig_jit);
tcase_add_test(tc_cli_arith, test_inf_jit);
tcase_add_test(tc_cli_arith, test_matchwithread_jit);
tcase_add_test(tc_cli_arith, test_pdf_jit);
tcase_add_test(tc_cli_arith, test_bswap_jit);
tcase_add_test(tc_cli_arith, test_inflate_jit);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
Separate bytecode tests into jit and interpreter. This makes it easier to see which one has a problem, and also reduces the runtime of individual tests.
2010-03-29 12:09:07 +03:00
tcase_add_test(tc_cli_arith, test_arith_int);
tcase_add_test(tc_cli_arith, test_apicalls_int);
tcase_add_test(tc_cli_arith, test_apicalls2_int);
tcase_add_test(tc_cli_arith, test_div0_int);
tcase_add_test(tc_cli_arith, test_lsig_int);
tcase_add_test(tc_cli_arith, test_inf_int);
tcase_add_test(tc_cli_arith, test_matchwithread_int);
tcase_add_test(tc_cli_arith, test_pdf_int);
tcase_add_test(tc_cli_arith, test_bswap_int);
tcase_add_test(tc_cli_arith, test_inflate_int);
fix compiler warnings (bb #1872, bb #1934, bb #1935)
2010-04-13 16:19:47 +03:00
tcase_add_test(tc_cli_arith, test_retmagic_int);
Add new bytecode API unit tests.
2010-05-13 22:44:29 +03:00
tcase_add_test(tc_cli_arith, test_api_extract_jit);
tcase_add_test(tc_cli_arith, test_api_files_jit);
tcase_add_test(tc_cli_arith, test_apicalls2_7_jit);
tcase_add_test(tc_cli_arith, test_apicalls_7_jit);
tcase_add_test(tc_cli_arith, test_apicalls_7_jit);
tcase_add_test(tc_cli_arith, test_arith_7_jit);
tcase_add_test(tc_cli_arith, test_debug_jit);
tcase_add_test(tc_cli_arith, test_inf_7_jit);
tcase_add_test(tc_cli_arith, test_lsig_7_jit);
tcase_add_test(tc_cli_arith, test_retmagic_7_jit);
tcase_add_test(tc_cli_arith, test_testadt_jit);
tcase_add_test(tc_cli_arith, test_api_extract_int);
tcase_add_test(tc_cli_arith, test_api_files_int);
tcase_add_test(tc_cli_arith, test_apicalls2_7_int);
tcase_add_test(tc_cli_arith, test_apicalls_7_int);
tcase_add_test(tc_cli_arith, test_apicalls_7_int);
tcase_add_test(tc_cli_arith, test_arith_7_int);
tcase_add_test(tc_cli_arith, test_debug_int);
tcase_add_test(tc_cli_arith, test_inf_7_int);
tcase_add_test(tc_cli_arith, test_lsig_7_int);
tcase_add_test(tc_cli_arith, test_retmagic_int);
tcase_add_test(tc_cli_arith, test_testadt_int);
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
tcase_add_test(tc_cli_arith, test_load_bytecode_jit);
tcase_add_test(tc_cli_arith, test_load_bytecode_int);
Restrict usage of pthread_barrier even more. Don't want to add a configure test for it now...
2010-11-23 13:31:47 +02:00
#ifdef DO_BARRIER
Make cl_load thread safe (bb #2333). Parallel cl_load() crash (bb #2333). Reason is twofold: - cache.c had 2 'static' global variables, thus trying to initialize same cache from multiple threads - bytecode2llvm.cpp: something in LLVM 2.7 is crashing when loading in parallel Fix is to drop the 'static' on the variable (cache is per engine already). This also fixes a potential memory leak in clamd! The other part of the fix is to turn on the mutex around bytecode compilation always. We don't call cl_load in parallel, so this doesn't affect clamd, but some may need to call cl_load in parallel.
2010-11-04 21:14:14 +02:00
tcase_add_test(tc_cli_arith, test_parallel_load);
#endif
Add bytecode.cvd load test.
2010-05-14 17:16:50 +03:00
Add missing unit-test files.
2009-07-13 19:45:05 +03:00
return s;
}
Reference in a new issue Copy permalink