2003-07-29 15:48:06 +00:00
|
|
|
/*
|
2022-01-06 16:53:44 -08:00
|
|
|
* Copyright (C) 2013-2022 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
|
2013-12-05 15:09:19 -08:00
|
|
|
* Copyright (C) 2007-2013 Sourcefire, Inc.
|
2008-02-01 00:17:44 +00:00
|
|
|
*
|
2008-04-02 15:24:51 +00:00
|
|
|
* Authors: Tomasz Kojm
|
2003-07-29 15:48:06 +00:00
|
|
|
*
|
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
2007-03-31 20:31:04 +00:00
|
|
|
* it under the terms of the GNU General Public License version 2 as
|
|
|
|
|
* published by the Free Software Foundation.
|
2003-07-29 15:48:06 +00:00
|
|
|
*
|
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
|
*
|
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
|
* along with this program; if not, write to the Free Software
|
2006-04-09 19:59:28 +00:00
|
|
|
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
|
|
|
|
|
* MA 02110-1301, USA.
|
2003-07-29 15:48:06 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef __MATCHER_H
|
|
|
|
|
#define __MATCHER_H
|
|
|
|
|
|
2006-11-20 00:03:16 +00:00
|
|
|
#include <sys/types.h>
|
|
|
|
|
|
2003-07-29 15:48:06 +00:00
|
|
|
#include "clamav.h"
|
2006-01-15 15:46:36 +00:00
|
|
|
#include "filetypes.h"
|
2006-02-15 00:41:40 +00:00
|
|
|
#include "others.h"
|
2006-11-20 00:03:16 +00:00
|
|
|
#include "execs.h"
|
2003-07-29 15:48:06 +00:00
|
|
|
|
2010-06-18 15:41:39 +02:00
|
|
|
struct cli_target_info {
|
|
|
|
|
off_t fsize;
|
|
|
|
|
struct cli_exe_info exeinfo;
|
|
|
|
|
int status; /* 0 == not initialised, 1 == initialised OK, -1 == error */
|
|
|
|
|
};
|
|
|
|
|
|
libclamav: Fix scan recursion tracking
Scan recursion is the process of identifying files embedded in other
files and then scanning them, recursively.
Internally this process is more complex than it may sound because a file
may have multiple layers of types before finding a new "file".
At present we treat the recursion count in the scanning context as an
index into both our fmap list AND our container list. These two lists
are conceptually a part of the same thing and should be unified.
But what's concerning is that the "recursion level" isn't actually
incremented or decremented at the same time that we add a layer to the
fmap or container lists but instead is more touchy-feely, increasing
when we find a new "file".
To account for this shadiness, the size of the fmap and container lists
has always been a little longer than our "max scan recursion" limit so
we don't accidentally overflow the fmap or container arrays (!).
I've implemented a single recursion-stack as an array, similar to before,
which includes a pointer to each fmap at each layer, along with the size
and type. Push and pop functions add and remove layers whenever a new
fmap is added. A boolean argument when pushing indicates if the new layer
represents a new buffer or new file (descriptor). A new buffer will reset
the "nested fmap level" (described below).
This commit also provides a solution for an issue where we detect
embedded files more than once during scan recursion.
For illustration, imagine a tarball named foo.tar.gz with this structure:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| baz.exe | PE | 0 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| │ └── hello.txt | ASCII | 2 | 0 |
| └── sfx.7z | 7Z | 1 | 1 |
| └── world.txt | ASCII | 2 | 0 |
(A) If we scan for embedded files at any layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| ├── foo.tar | TAR | 1 | 0 |
| │ ├── bar.zip | ZIP | 2 | 1 |
| │ │ └── hola.txt | ASCII | 3 | 0 |
| │ ├── baz.exe | PE | 2 | 1 |
| │ │ ├── sfx.zip | ZIP | 3 | 1 |
| │ │ │ └── hello.txt | ASCII | 4 | 0 |
| │ │ └── sfx.7z | 7Z | 3 | 1 |
| │ │ └── world.txt | ASCII | 4 | 0 |
| │ ├── sfx.zip | ZIP | 2 | 1 |
| │ │ └── hello.txt | ASCII | 3 | 0 |
| │ └── sfx.7z | 7Z | 2 | 1 |
| │ └── world.txt | ASCII | 3 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| └── sfx.7z | 7Z | 1 | 1 |
(A) is bad because it scans content more than once.
Note that for the GZ layer, it may detect the ZIP and 7Z if the
signature hits on the compressed data, which it might, though
extracting the ZIP and 7Z will likely fail.
The reason the above doesn't happen now is that we restrict embedded
type scans for a bunch of archive formats to include GZ and TAR.
(B) If we scan for embedded files at the foo.tar layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| ├── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 2 | 1 |
| │ └── hello.txt | ASCII | 3 | 0 |
| └── sfx.7z | 7Z | 2 | 1 |
| └── world.txt | ASCII | 3 | 0 |
(B) is almost right. But we can achieve it easily enough only scanning for
embedded content in the current fmap when the "nested fmap level" is 0.
The upside is that it should safely detect all embedded content, even if
it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe.
The biggest risk I can think of affects ZIPs. SFXZIP detection
is identical to ZIP detection, which is why we don't allow SFXZIP to be
detected if insize of a ZIP. If we only allow embedded type scanning at
fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP
if the bar.exe was not compressed in foo.zip and if non-compressed files
extracted from ZIPs aren't extracted as new buffers:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.zip | ZIP | 0 | 0 |
| └── bar.exe | PE | 1 | 1 |
| └── sfx.zip | ZIP | 2 | 2 |
Provided that we ensure all files extracted from zips are scanned in
new buffers, option (B) should be safe.
(C) If we scan for embedded files at the baz.exe layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 3 | 1 |
| │ └── hello.txt | ASCII | 4 | 0 |
| └── sfx.7z | 7Z | 3 | 1 |
| └── world.txt | ASCII | 4 | 0 |
(C) is right. But it's harder to achieve. For this example we can get it by
restricting 7ZSFX and ZIPSFX detection only when scanning an executable.
But that may mean losing detection of archives embedded elsewhere.
And we'd have to identify allowable container types for each possible
embedded type, which would be very difficult.
So this commit aims to solve the issue the (B)-way.
Note that in all situations, we still have to scan with file typing
enabled to determine if we need to reassign the current file type, such
as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2-
compressed. Detection of DMG and a handful of other types rely on
finding data partway through or near the ned of a file before
reassigning the entire file as the new type.
Other fixes and considerations in this commit:
- The utf16 HTML parser has weak error handling, particularly with respect
to creating a nested fmap for scanning the ascii decoded file.
This commit cleans up the error handling and wraps the nested scan with
the recursion-stack push()/pop() for correct recursion tracking.
Before this commit, each container layer had a flag to indicate if the
container layer is valid.
We need something similar so that the cli_recursion_stack_get_*()
functions ignore normalized layers. Details...
Imagine an LDB signature for HTML content that specifies a ZIP
container. If the signature actually alerts on the normalized HTML and
you don't ignore normalized layers for the container check, it will
appear as though the alert is in an HTML container rather than a ZIP
container.
This commit accomplishes this with a boolean you set in the scan context
before scanning a new layer. Then when the new fmap is created, it will
use that flag to set similar flag for the layer. The context flag is
reset those that anything after this doesn't have that flag.
The flag allows the new recursion_stack_get() function to ignore
normalized layers when iterating the stack to return a layer at a
requested index, negative or positive.
Scanning normalized extracted/normalized javascript and VBA should also
use the 'layer is normalized' flag.
- This commit also fixes Heuristic.Broken.Executable alert for ELF files
to make sure that:
A) these only alert if cli_append_virus() returns CL_VIRUS (aka it
respects the FP check).
B) all broken-executable alerts for ELF only happen if the
SCAN_HEURISTIC_BROKEN option is enabled.
- This commit also cleans up the error handling in cli_magic_scan_dir().
This was needed so we could correctly apply the layer-is-normalized-flag
to all VBA macros extracted to a directory when scanning the directory.
- Also fix an issue where exceeding scan maximums wouldn't cause embedded
file detection scans to abort. Granted we don't actually want to abort
if max filesize or max recursion depth are exceeded... only if max
scansize, max files, and max scantime are exceeded.
Add 'abort_scan' flag to scan context, to protect against depending on
correct error propagation for fatal conditions. Instead, setting this
flag in the scan context should guarantee that a fatal condition deep in
scan recursion isn't lost which result in more stuff being scanned
instead of aborting. This shouldn't be necessary, but some status codes
like CL_ETIMEOUT never used to be fatal and it's easier to do this than
to verify every parser only returns CL_ETIMEOUT and other "fatal
status codes" in fatal conditions.
- Remove duplicate is_tar() prototype from filestypes.c and include
is_tar.h instead.
- Presently we create the fmap hash when creating the fmap.
This wastes a bit of CPU if the hash is never needed.
Now that we're creating fmap's for all embedded files discovered with
file type recognition scans, this is a much more frequent occurence and
really slows things down.
This commit fixes the issue by only creating fmap hashes as needed.
This should not only resolve the perfomance impact of creating fmap's
for all embedded files, but also should improve performance in general.
- Add allmatch check to the zip parser after the central-header meta
match. That way we don't multiple alerts with the same match except in
allmatch mode. Clean up error handling in the zip parser a tiny bit.
- Fixes to ensure that the scan limits such as scansize, filesize,
recursion depth, # of embedded files, and scantime are always reported
if AlertExceedsMax (--alert-exceeds-max) is enabled.
- Fixed an issue where non-fatal alerts for exceeding scan maximums may
mask signature matches later on. I changed it so these alerts use the
"possibly unwanted" alert-type and thus only alert if no other alerts
were found or if all-match or heuristic-precedence are enabled.
- Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata
when the --gen-json feature is enabled. These will show up once under
"ParseErrors" the first time a limit is exceeded. In the present
implementation, only one limits-exceeded events will be added, so as to
prevent a malicious or malformed sample from filling the JSON buffer
with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
|
|
|
/**
|
|
|
|
|
* Initialize a struct cli_target_info so that it's ready to have its exeinfo
|
|
|
|
|
* populated by the call to cli_targetinfo and/or destroyed by
|
|
|
|
|
* cli_targetinfo_destroy.
|
|
|
|
|
*
|
|
|
|
|
* @param info a pointer to the struct cli_target_info to initialize
|
|
|
|
|
*/
|
PE parsing code improvements, db loading bug fixes
Consolidate the PE parsing code into one function. I tried to preserve all existing functionality from the previous, distinct implementations to a large extent (with the exceptions mentioned below). If I noticed potential bugs/improvements, I added a TODO statement about those so that they can be fixed in a smaller commit later. Also, there are more TODOs in places where I'm not entirely sure why certain actions are performed - more research is needed for these.
I'm submitting a pull request now so that regression testing can be done, and because merging what I have thus far now will likely have fewer conflicts than if I try to merge later
PE parsing code improvements:
- PEs without all 16 data directories are parsed more appropriately now
- Added lots more debug statements
Also:
- Allow MAX_BC and MAX_TRACKED_PCRE to be specified via CFLAGS
When doing performance testing with the latest CVD, MAX_BC and
MAX_TRACKED_PCRE need to be raised to track all the events.
Allow these to be specified via CFLAGS by not redefining them
if they are already defined
- Fix an issue preventing wildcard sizes in .MDB/.MSB rules
I'm not sure what the original intent of the check I removed was,
but it prevents using wildcard sizes in .MDB/.MSB rules. AFAICT
these wildcard sizes should be handled appropriately by the MD5
section hash computation code, so I don't think a check on that
is needed.
- Fix several issues related to db loading
- .imp files will now get loaded if they exist in a directory passed
via clamscan's '-d' flag
- .pwdb files will now get loaded if they exist in a directory passed
via clamscan's '-d' flag even when compiling without yara support
- Changes to .imp, .ign, and .ign2 files will now be reflected in calls
to cl_statinidir and cl_statchkdir (and also .pwdb files, even when
compiling without yara support)
- The contents of .sfp files won't be included in some of the signature
counts, and the contents of .cud files will be
- Any local.gdb files will no longer be loaded twice
- For .imp files, you are no longer required to specify a minimum flevel for wildcard rules, since this isn't needed
2019-01-08 00:09:08 -05:00
|
|
|
void cli_targetinfo_init(struct cli_target_info *info);
|
libclamav: Fix scan recursion tracking
Scan recursion is the process of identifying files embedded in other
files and then scanning them, recursively.
Internally this process is more complex than it may sound because a file
may have multiple layers of types before finding a new "file".
At present we treat the recursion count in the scanning context as an
index into both our fmap list AND our container list. These two lists
are conceptually a part of the same thing and should be unified.
But what's concerning is that the "recursion level" isn't actually
incremented or decremented at the same time that we add a layer to the
fmap or container lists but instead is more touchy-feely, increasing
when we find a new "file".
To account for this shadiness, the size of the fmap and container lists
has always been a little longer than our "max scan recursion" limit so
we don't accidentally overflow the fmap or container arrays (!).
I've implemented a single recursion-stack as an array, similar to before,
which includes a pointer to each fmap at each layer, along with the size
and type. Push and pop functions add and remove layers whenever a new
fmap is added. A boolean argument when pushing indicates if the new layer
represents a new buffer or new file (descriptor). A new buffer will reset
the "nested fmap level" (described below).
This commit also provides a solution for an issue where we detect
embedded files more than once during scan recursion.
For illustration, imagine a tarball named foo.tar.gz with this structure:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| baz.exe | PE | 0 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| │ └── hello.txt | ASCII | 2 | 0 |
| └── sfx.7z | 7Z | 1 | 1 |
| └── world.txt | ASCII | 2 | 0 |
(A) If we scan for embedded files at any layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| ├── foo.tar | TAR | 1 | 0 |
| │ ├── bar.zip | ZIP | 2 | 1 |
| │ │ └── hola.txt | ASCII | 3 | 0 |
| │ ├── baz.exe | PE | 2 | 1 |
| │ │ ├── sfx.zip | ZIP | 3 | 1 |
| │ │ │ └── hello.txt | ASCII | 4 | 0 |
| │ │ └── sfx.7z | 7Z | 3 | 1 |
| │ │ └── world.txt | ASCII | 4 | 0 |
| │ ├── sfx.zip | ZIP | 2 | 1 |
| │ │ └── hello.txt | ASCII | 3 | 0 |
| │ └── sfx.7z | 7Z | 2 | 1 |
| │ └── world.txt | ASCII | 3 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| └── sfx.7z | 7Z | 1 | 1 |
(A) is bad because it scans content more than once.
Note that for the GZ layer, it may detect the ZIP and 7Z if the
signature hits on the compressed data, which it might, though
extracting the ZIP and 7Z will likely fail.
The reason the above doesn't happen now is that we restrict embedded
type scans for a bunch of archive formats to include GZ and TAR.
(B) If we scan for embedded files at the foo.tar layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| ├── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 2 | 1 |
| │ └── hello.txt | ASCII | 3 | 0 |
| └── sfx.7z | 7Z | 2 | 1 |
| └── world.txt | ASCII | 3 | 0 |
(B) is almost right. But we can achieve it easily enough only scanning for
embedded content in the current fmap when the "nested fmap level" is 0.
The upside is that it should safely detect all embedded content, even if
it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe.
The biggest risk I can think of affects ZIPs. SFXZIP detection
is identical to ZIP detection, which is why we don't allow SFXZIP to be
detected if insize of a ZIP. If we only allow embedded type scanning at
fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP
if the bar.exe was not compressed in foo.zip and if non-compressed files
extracted from ZIPs aren't extracted as new buffers:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.zip | ZIP | 0 | 0 |
| └── bar.exe | PE | 1 | 1 |
| └── sfx.zip | ZIP | 2 | 2 |
Provided that we ensure all files extracted from zips are scanned in
new buffers, option (B) should be safe.
(C) If we scan for embedded files at the baz.exe layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 3 | 1 |
| │ └── hello.txt | ASCII | 4 | 0 |
| └── sfx.7z | 7Z | 3 | 1 |
| └── world.txt | ASCII | 4 | 0 |
(C) is right. But it's harder to achieve. For this example we can get it by
restricting 7ZSFX and ZIPSFX detection only when scanning an executable.
But that may mean losing detection of archives embedded elsewhere.
And we'd have to identify allowable container types for each possible
embedded type, which would be very difficult.
So this commit aims to solve the issue the (B)-way.
Note that in all situations, we still have to scan with file typing
enabled to determine if we need to reassign the current file type, such
as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2-
compressed. Detection of DMG and a handful of other types rely on
finding data partway through or near the ned of a file before
reassigning the entire file as the new type.
Other fixes and considerations in this commit:
- The utf16 HTML parser has weak error handling, particularly with respect
to creating a nested fmap for scanning the ascii decoded file.
This commit cleans up the error handling and wraps the nested scan with
the recursion-stack push()/pop() for correct recursion tracking.
Before this commit, each container layer had a flag to indicate if the
container layer is valid.
We need something similar so that the cli_recursion_stack_get_*()
functions ignore normalized layers. Details...
Imagine an LDB signature for HTML content that specifies a ZIP
container. If the signature actually alerts on the normalized HTML and
you don't ignore normalized layers for the container check, it will
appear as though the alert is in an HTML container rather than a ZIP
container.
This commit accomplishes this with a boolean you set in the scan context
before scanning a new layer. Then when the new fmap is created, it will
use that flag to set similar flag for the layer. The context flag is
reset those that anything after this doesn't have that flag.
The flag allows the new recursion_stack_get() function to ignore
normalized layers when iterating the stack to return a layer at a
requested index, negative or positive.
Scanning normalized extracted/normalized javascript and VBA should also
use the 'layer is normalized' flag.
- This commit also fixes Heuristic.Broken.Executable alert for ELF files
to make sure that:
A) these only alert if cli_append_virus() returns CL_VIRUS (aka it
respects the FP check).
B) all broken-executable alerts for ELF only happen if the
SCAN_HEURISTIC_BROKEN option is enabled.
- This commit also cleans up the error handling in cli_magic_scan_dir().
This was needed so we could correctly apply the layer-is-normalized-flag
to all VBA macros extracted to a directory when scanning the directory.
- Also fix an issue where exceeding scan maximums wouldn't cause embedded
file detection scans to abort. Granted we don't actually want to abort
if max filesize or max recursion depth are exceeded... only if max
scansize, max files, and max scantime are exceeded.
Add 'abort_scan' flag to scan context, to protect against depending on
correct error propagation for fatal conditions. Instead, setting this
flag in the scan context should guarantee that a fatal condition deep in
scan recursion isn't lost which result in more stuff being scanned
instead of aborting. This shouldn't be necessary, but some status codes
like CL_ETIMEOUT never used to be fatal and it's easier to do this than
to verify every parser only returns CL_ETIMEOUT and other "fatal
status codes" in fatal conditions.
- Remove duplicate is_tar() prototype from filestypes.c and include
is_tar.h instead.
- Presently we create the fmap hash when creating the fmap.
This wastes a bit of CPU if the hash is never needed.
Now that we're creating fmap's for all embedded files discovered with
file type recognition scans, this is a much more frequent occurence and
really slows things down.
This commit fixes the issue by only creating fmap hashes as needed.
This should not only resolve the perfomance impact of creating fmap's
for all embedded files, but also should improve performance in general.
- Add allmatch check to the zip parser after the central-header meta
match. That way we don't multiple alerts with the same match except in
allmatch mode. Clean up error handling in the zip parser a tiny bit.
- Fixes to ensure that the scan limits such as scansize, filesize,
recursion depth, # of embedded files, and scantime are always reported
if AlertExceedsMax (--alert-exceeds-max) is enabled.
- Fixed an issue where non-fatal alerts for exceeding scan maximums may
mask signature matches later on. I changed it so these alerts use the
"possibly unwanted" alert-type and thus only alert if no other alerts
were found or if all-match or heuristic-precedence are enabled.
- Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata
when the --gen-json feature is enabled. These will show up once under
"ParseErrors" the first time a limit is exceeded. In the present
implementation, only one limits-exceeded events will be added, so as to
prevent a malicious or malformed sample from filling the JSON buffer
with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Free resources associated with a struct cli_target_info initialized
|
|
|
|
|
* via cli_targetinfo_init
|
|
|
|
|
*
|
|
|
|
|
* @param info a pointer to the struct cli_target_info to destroy
|
|
|
|
|
*/
|
PE parsing code improvements, db loading bug fixes
Consolidate the PE parsing code into one function. I tried to preserve all existing functionality from the previous, distinct implementations to a large extent (with the exceptions mentioned below). If I noticed potential bugs/improvements, I added a TODO statement about those so that they can be fixed in a smaller commit later. Also, there are more TODOs in places where I'm not entirely sure why certain actions are performed - more research is needed for these.
I'm submitting a pull request now so that regression testing can be done, and because merging what I have thus far now will likely have fewer conflicts than if I try to merge later
PE parsing code improvements:
- PEs without all 16 data directories are parsed more appropriately now
- Added lots more debug statements
Also:
- Allow MAX_BC and MAX_TRACKED_PCRE to be specified via CFLAGS
When doing performance testing with the latest CVD, MAX_BC and
MAX_TRACKED_PCRE need to be raised to track all the events.
Allow these to be specified via CFLAGS by not redefining them
if they are already defined
- Fix an issue preventing wildcard sizes in .MDB/.MSB rules
I'm not sure what the original intent of the check I removed was,
but it prevents using wildcard sizes in .MDB/.MSB rules. AFAICT
these wildcard sizes should be handled appropriately by the MD5
section hash computation code, so I don't think a check on that
is needed.
- Fix several issues related to db loading
- .imp files will now get loaded if they exist in a directory passed
via clamscan's '-d' flag
- .pwdb files will now get loaded if they exist in a directory passed
via clamscan's '-d' flag even when compiling without yara support
- Changes to .imp, .ign, and .ign2 files will now be reflected in calls
to cl_statinidir and cl_statchkdir (and also .pwdb files, even when
compiling without yara support)
- The contents of .sfp files won't be included in some of the signature
counts, and the contents of .cud files will be
- Any local.gdb files will no longer be loaded twice
- For .imp files, you are no longer required to specify a minimum flevel for wildcard rules, since this isn't needed
2019-01-08 00:09:08 -05:00
|
|
|
void cli_targetinfo_destroy(struct cli_target_info *info);
|
|
|
|
|
|
2007-03-28 21:38:07 +00:00
|
|
|
#include "matcher-ac.h"
|
|
|
|
|
#include "matcher-bm.h"
|
2011-01-07 02:59:41 +01:00
|
|
|
#include "matcher-hash.h"
|
2014-08-22 14:39:17 -04:00
|
|
|
#include "matcher-pcre.h"
|
2018-09-25 17:44:19 -04:00
|
|
|
#include "matcher-byte-comp.h"
|
2014-08-22 14:39:17 -04:00
|
|
|
#include "regex_pcre.h"
|
2009-09-01 13:49:36 +02:00
|
|
|
#include "fmap.h"
|
2008-10-17 17:00:13 +00:00
|
|
|
#include "mpool.h"
|
|
|
|
|
|
2018-12-03 12:37:58 -05:00
|
|
|
// clang-format off
|
|
|
|
|
|
|
|
|
|
#define CLI_MATCH_METADATA 0xff00
|
|
|
|
|
#define CLI_MATCH_WILDCARD 0x0f00
|
|
|
|
|
#define CLI_MATCH_CHAR 0x0000
|
|
|
|
|
#define CLI_MATCH_NOCASE 0x1000
|
|
|
|
|
#define CLI_MATCH_IGNORE 0x0100
|
|
|
|
|
#define CLI_MATCH_SPECIAL 0x0200
|
|
|
|
|
#define CLI_MATCH_NIBBLE_HIGH 0x0300
|
|
|
|
|
#define CLI_MATCH_NIBBLE_LOW 0x0400
|
|
|
|
|
|
2020-03-19 21:23:54 -04:00
|
|
|
typedef enum tdb_type {
|
|
|
|
|
CLI_TDB_UINT,
|
|
|
|
|
CLI_TDB_RANGE,
|
|
|
|
|
CLI_TDB_STR,
|
|
|
|
|
CLI_TDB_RANGE2,
|
|
|
|
|
CLI_TDB_FTYPE,
|
|
|
|
|
CLI_TDB_FTYPE_EXPR
|
|
|
|
|
} tdb_type_t;
|
2007-03-28 21:38:07 +00:00
|
|
|
|
2008-07-25 19:00:25 +00:00
|
|
|
struct cli_lsig_tdb {
|
2018-12-03 12:37:58 -05:00
|
|
|
uint32_t *val, *range;
|
|
|
|
|
char *str;
|
2020-03-19 21:23:54 -04:00
|
|
|
tdb_type_t cnt[3];
|
2018-12-03 12:37:58 -05:00
|
|
|
uint32_t subsigs;
|
2008-07-25 19:00:25 +00:00
|
|
|
|
|
|
|
|
const uint32_t *target;
|
2009-12-30 00:20:02 +01:00
|
|
|
const uint32_t *engine, *nos, *ep, *filesize;
|
2010-08-24 12:28:16 +02:00
|
|
|
const uint32_t *container, *handlertype;
|
2017-01-23 13:11:03 -05:00
|
|
|
const uint32_t *intermediates;
|
2009-12-14 17:16:46 +01:00
|
|
|
/*
|
2008-07-25 19:00:25 +00:00
|
|
|
const uint32_t *sectoff, *sectrva, *sectvsz, *sectraw, *sectrsz,
|
2018-12-03 12:37:58 -05:00
|
|
|
*secturva, *sectuvsz, *secturaw, *sectursz;
|
2009-12-14 17:16:46 +01:00
|
|
|
*/
|
2018-12-03 12:37:58 -05:00
|
|
|
const char *icongrp1, *icongrp2;
|
|
|
|
|
uint32_t *macro_ptids;
|
2008-10-19 16:16:49 +00:00
|
|
|
#ifdef USE_MPOOL
|
2018-12-03 12:37:58 -05:00
|
|
|
mpool_t *mempool;
|
2021-04-21 16:24:24 -07:00
|
|
|
#else
|
|
|
|
|
void *_padding_mempool;
|
2008-10-19 16:16:49 +00:00
|
|
|
#endif
|
2008-07-25 19:00:25 +00:00
|
|
|
};
|
|
|
|
|
|
2018-12-03 12:37:58 -05:00
|
|
|
// clang-format on
|
|
|
|
|
|
2015-06-19 16:33:59 -04:00
|
|
|
#define CLI_LSIG_FLAG_PRIVATE 0x01
|
|
|
|
|
|
2020-03-19 21:23:54 -04:00
|
|
|
typedef enum lsig_type {
|
|
|
|
|
CLI_LSIG_NORMAL,
|
|
|
|
|
CLI_YARA_NORMAL,
|
|
|
|
|
CLI_YARA_OFFSET
|
|
|
|
|
} lsig_type_t;
|
|
|
|
|
|
2009-09-21 23:44:32 +03:00
|
|
|
struct cli_bc;
|
2008-07-25 19:00:25 +00:00
|
|
|
struct cli_ac_lsig {
|
|
|
|
|
uint32_t id;
|
2010-06-09 13:42:47 +03:00
|
|
|
unsigned bc_idx;
|
2020-03-19 21:23:54 -04:00
|
|
|
lsig_type_t type;
|
2015-06-19 16:33:59 -04:00
|
|
|
uint8_t flag;
|
2015-02-11 10:36:43 -08:00
|
|
|
union {
|
|
|
|
|
char *logic;
|
2015-03-06 17:10:47 -05:00
|
|
|
uint8_t *code_start;
|
2015-02-11 10:36:43 -08:00
|
|
|
} u;
|
2021-04-21 16:24:24 -07:00
|
|
|
char *virname;
|
2008-07-25 19:00:25 +00:00
|
|
|
struct cli_lsig_tdb tdb;
|
|
|
|
|
};
|
|
|
|
|
|
2022-02-28 15:41:28 -08:00
|
|
|
typedef void *fuzzyhashmap_t;
|
2021-04-21 16:24:24 -07:00
|
|
|
|
2007-03-28 21:38:07 +00:00
|
|
|
struct cli_matcher {
|
2009-08-14 14:38:13 +02:00
|
|
|
unsigned int type;
|
|
|
|
|
|
2007-03-28 21:38:07 +00:00
|
|
|
/* Extended Boyer-Moore */
|
2007-11-06 17:16:07 +00:00
|
|
|
uint8_t *bm_shift;
|
2009-09-01 11:19:31 +02:00
|
|
|
struct cli_bm_patt **bm_suffix, **bm_pattab;
|
2007-04-30 14:12:38 +00:00
|
|
|
uint32_t *soff, soff_len; /* for PE section sigs */
|
2009-09-01 11:19:31 +02:00
|
|
|
uint32_t bm_offmode, bm_patterns, bm_reloff_num, bm_absoff_num;
|
2007-03-28 21:38:07 +00:00
|
|
|
|
2011-01-14 16:12:43 +01:00
|
|
|
/* HASH */
|
2011-01-07 02:59:41 +01:00
|
|
|
struct cli_hash_patt hm;
|
2013-03-08 18:10:07 -05:00
|
|
|
struct cli_hash_wild hwild;
|
2011-01-07 02:59:41 +01:00
|
|
|
|
2007-03-28 21:38:07 +00:00
|
|
|
/* Extended Aho-Corasick */
|
2015-02-10 09:23:51 -08:00
|
|
|
uint32_t ac_partsigs, ac_nodes, ac_lists, ac_patterns, ac_lsigs;
|
2008-07-25 19:00:25 +00:00
|
|
|
struct cli_ac_lsig **ac_lsigtable;
|
2007-03-28 21:38:07 +00:00
|
|
|
struct cli_ac_node *ac_root, **ac_nodetable;
|
2015-02-10 09:23:51 -08:00
|
|
|
struct cli_ac_list **ac_listtable;
|
2007-04-28 18:40:59 +00:00
|
|
|
struct cli_ac_patt **ac_pattable;
|
2009-08-14 14:38:13 +02:00
|
|
|
struct cli_ac_patt **ac_reloff;
|
2009-08-21 15:55:10 +02:00
|
|
|
uint32_t ac_reloff_num, ac_absoff_num;
|
2008-01-24 13:24:02 +00:00
|
|
|
uint8_t ac_mindepth, ac_maxdepth;
|
2010-02-10 11:39:47 +02:00
|
|
|
struct filter *filter;
|
2008-01-24 13:24:02 +00:00
|
|
|
|
|
|
|
|
uint16_t maxpatlen;
|
2015-02-09 14:22:45 -08:00
|
|
|
uint8_t ac_only;
|
2014-08-22 14:39:17 -04:00
|
|
|
|
|
|
|
|
/* Perl-Compiled Regular Expressions */
|
2014-08-25 15:07:30 -04:00
|
|
|
#if HAVE_PCRE
|
2014-08-29 14:57:09 -04:00
|
|
|
uint32_t pcre_metas;
|
|
|
|
|
struct cli_pcre_meta **pcre_metatable;
|
2014-09-09 17:14:12 -04:00
|
|
|
uint32_t pcre_reloff_num, pcre_absoff_num;
|
2014-08-25 15:07:30 -04:00
|
|
|
#endif
|
2014-08-22 14:39:17 -04:00
|
|
|
|
2018-09-21 16:49:38 -04:00
|
|
|
/* Byte Compare */
|
|
|
|
|
uint32_t bcomp_metas;
|
|
|
|
|
struct cli_bcomp_meta **bcomp_metatable;
|
|
|
|
|
|
2021-04-21 16:24:24 -07:00
|
|
|
/* Fuzzy Image Hash */
|
2022-02-28 15:41:28 -08:00
|
|
|
fuzzyhashmap_t fuzzy_hashmap;
|
2021-04-21 16:24:24 -07:00
|
|
|
|
2016-06-28 15:18:30 -04:00
|
|
|
/* Bytecode Tracker */
|
|
|
|
|
uint32_t linked_bcs;
|
|
|
|
|
|
2022-09-30 10:43:55 -07:00
|
|
|
/*Store pointers to malloced trans values so that they can be more easily freed*/
|
2022-10-18 09:54:50 -07:00
|
|
|
struct cli_ac_node ***trans_array;
|
2022-09-30 10:43:55 -07:00
|
|
|
size_t trans_cnt;
|
|
|
|
|
size_t trans_capacity;
|
|
|
|
|
|
2008-10-18 00:16:23 +00:00
|
|
|
#ifdef USE_MPOOL
|
2009-01-26 19:47:02 +00:00
|
|
|
mpool_t *mempool;
|
2021-04-21 16:24:24 -07:00
|
|
|
#else
|
|
|
|
|
void *_padding_mempool;
|
2008-10-18 00:16:23 +00:00
|
|
|
#endif
|
2007-03-28 21:38:07 +00:00
|
|
|
};
|
|
|
|
|
|
2018-12-03 12:40:13 -05:00
|
|
|
struct cli_cdb {
|
|
|
|
|
char *virname; /* virus name */
|
|
|
|
|
cli_file_t ctype; /* container type */
|
|
|
|
|
regex_t name; /* filename regex */
|
|
|
|
|
size_t csize[2]; /* container size (min, max); if csize[0] != csize[1]
|
GIF, PNG bugfixes; Add AlertBrokenMedia option
Added a new scan option to alert on broken media (graphics) file
formats. This feature mitigates the risk of malformed media files
intended to exploit vulnerabilities in other software. At present
media validation exists for JPEG, TIFF, PNG, and GIF files.
To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or
use the `--alert-broken-media` option when using `clamscan`.
These options are disabled by default for now.
Application developers may enable this scan option by enabling
`CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit
field.
Fixed PNG parser logic bugs that caused an excess of parsing errors
and fixed a stack exhaustion issue affecting some systems when
scanning PNG files. PNG file type detection was disabled via
signature database update for 0.103.0 to mitigate effects from these
bugs.
Fixed an issue where PNG and GIF files no longer work with Target:5
(graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as
CL_TYPE_GRAPHICS. Target types now support up to 10 possible file
types to make way for additional graphics types in future releases.
Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse"
errors when file format validation fails. Instead, the scan will alert
with the "Heuristics.Broken.Media" signature prefix and a descriptive
suffix to indicate the issue, provided that the "alert broken media"
feature is enabled.
GIF format validation will no longer fail if the GIF image is missing
the trailer byte, as this appears to be a relatively common issue in
otherwise functional GIF files.
Added a TIFF dynamic configuration (DCONF) option, which was missing.
This will allow us to disable TIFF format validation via signature
database update in the event that it proves to be problematic.
This feature already exists for many other file types.
Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
2020-11-04 15:49:43 -08:00
|
|
|
* then value of 0 makes the field ignored
|
|
|
|
|
*/
|
2018-12-03 12:40:13 -05:00
|
|
|
size_t fsizec[2]; /* file size in container */
|
|
|
|
|
size_t fsizer[2]; /* real file size */
|
|
|
|
|
int encrypted; /* file is encrypted; 2 == ignore */
|
|
|
|
|
unsigned int filepos[2]; /* file position in container */
|
|
|
|
|
int res1; /* reserved / format specific */
|
|
|
|
|
void *res2; /* reserved / format specific */
|
2010-01-07 18:26:12 +01:00
|
|
|
|
|
|
|
|
struct cli_cdb *next;
|
|
|
|
|
};
|
|
|
|
|
|
GIF, PNG bugfixes; Add AlertBrokenMedia option
Added a new scan option to alert on broken media (graphics) file
formats. This feature mitigates the risk of malformed media files
intended to exploit vulnerabilities in other software. At present
media validation exists for JPEG, TIFF, PNG, and GIF files.
To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or
use the `--alert-broken-media` option when using `clamscan`.
These options are disabled by default for now.
Application developers may enable this scan option by enabling
`CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit
field.
Fixed PNG parser logic bugs that caused an excess of parsing errors
and fixed a stack exhaustion issue affecting some systems when
scanning PNG files. PNG file type detection was disabled via
signature database update for 0.103.0 to mitigate effects from these
bugs.
Fixed an issue where PNG and GIF files no longer work with Target:5
(graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as
CL_TYPE_GRAPHICS. Target types now support up to 10 possible file
types to make way for additional graphics types in future releases.
Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse"
errors when file format validation fails. Instead, the scan will alert
with the "Heuristics.Broken.Media" signature prefix and a descriptive
suffix to indicate the issue, provided that the "alert broken media"
feature is enabled.
GIF format validation will no longer fail if the GIF image is missing
the trailer byte, as this appears to be a relatively common issue in
otherwise functional GIF files.
Added a TIFF dynamic configuration (DCONF) option, which was missing.
This will allow us to disable TIFF format validation via signature
database update in the event that it proves to be problematic.
This feature already exists for many other file types.
Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
2020-11-04 15:49:43 -08:00
|
|
|
#define CLI_MAX_TARGETS 10 /* maximum filetypes for a specific target */
|
2008-02-01 00:17:44 +00:00
|
|
|
struct cli_mtarget {
|
2014-02-21 16:10:32 -05:00
|
|
|
cli_file_t target[CLI_MAX_TARGETS];
|
2008-02-01 00:17:44 +00:00
|
|
|
const char *name;
|
2018-12-03 12:40:13 -05:00
|
|
|
uint8_t idx; /* idx of matcher */
|
2008-02-01 00:17:44 +00:00
|
|
|
uint8_t ac_only;
|
2010-02-10 11:39:47 +02:00
|
|
|
uint8_t enable_prefiltering;
|
2014-02-21 16:10:32 -05:00
|
|
|
uint8_t target_count; /* must be synced with non-zero values in the target array */
|
2008-02-01 00:17:44 +00:00
|
|
|
};
|
|
|
|
|
|
2015-07-23 16:37:15 -04:00
|
|
|
#define CLI_MTARGETS 15
|
GIF, PNG bugfixes; Add AlertBrokenMedia option
Added a new scan option to alert on broken media (graphics) file
formats. This feature mitigates the risk of malformed media files
intended to exploit vulnerabilities in other software. At present
media validation exists for JPEG, TIFF, PNG, and GIF files.
To enable this feature, set `AlertBrokenMedia yes` in clamd.conf, or
use the `--alert-broken-media` option when using `clamscan`.
These options are disabled by default for now.
Application developers may enable this scan option by enabling
`CL_SCAN_HEURISTIC_BROKEN_MEDIA` for the `heuristic` scan option bit
field.
Fixed PNG parser logic bugs that caused an excess of parsing errors
and fixed a stack exhaustion issue affecting some systems when
scanning PNG files. PNG file type detection was disabled via
signature database update for 0.103.0 to mitigate effects from these
bugs.
Fixed an issue where PNG and GIF files no longer work with Target:5
(graphics) signatures if detected as CL_TYPE_PNG/GIF rather than as
CL_TYPE_GRAPHICS. Target types now support up to 10 possible file
types to make way for additional graphics types in future releases.
Scanning JPEG, TIFF, PNG, and GIF files will no longer return "parse"
errors when file format validation fails. Instead, the scan will alert
with the "Heuristics.Broken.Media" signature prefix and a descriptive
suffix to indicate the issue, provided that the "alert broken media"
feature is enabled.
GIF format validation will no longer fail if the GIF image is missing
the trailer byte, as this appears to be a relatively common issue in
otherwise functional GIF files.
Added a TIFF dynamic configuration (DCONF) option, which was missing.
This will allow us to disable TIFF format validation via signature
database update in the event that it proves to be problematic.
This feature already exists for many other file types.
Added CL_TYPE_JPEG and CL_TYPE_TIFF types.
2020-11-04 15:49:43 -08:00
|
|
|
static const struct cli_mtarget cli_mtargets[CLI_MTARGETS] = {
|
|
|
|
|
/* All types for target, name, idx, ac_only, pre-filtering?, # of types */
|
|
|
|
|
{{0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "GENERIC", 0, 0, 1, 1},
|
|
|
|
|
{{CL_TYPE_MSEXE, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "PE", 1, 0, 1, 1},
|
|
|
|
|
{{CL_TYPE_MSOLE2, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "OLE2", 2, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_HTML, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "HTML", 3, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_MAIL, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "MAIL", 4, 1, 1, 1},
|
|
|
|
|
{{CL_TYPE_GRAPHICS, CL_TYPE_GIF, CL_TYPE_PNG, CL_TYPE_JPEG, CL_TYPE_TIFF, 0, 0, 0, 0, 0}, "GRAPHICS", 5, 1, 0, 5},
|
|
|
|
|
{{CL_TYPE_ELF, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "ELF", 6, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_TEXT_ASCII, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "ASCII", 7, 1, 1, 1},
|
|
|
|
|
{{CL_TYPE_ERROR, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "NOT USED", 8, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_MACHO, CL_TYPE_MACHO_UNIBIN, 0, 0, 0, 0, 0, 0, 0, 0}, "MACH-O", 9, 1, 0, 2},
|
|
|
|
|
{{CL_TYPE_PDF, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "PDF", 10, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_SWF, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "FLASH", 11, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_JAVA, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "JAVA", 12, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_INTERNAL, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "INTERNAL", 13, 1, 0, 1},
|
|
|
|
|
{{CL_TYPE_OTHER, 0, 0, 0, 0, 0, 0, 0, 0, 0}, "OTHER", 14, 1, 0, 1}};
|
|
|
|
|
|
|
|
|
|
// clang-format off
|
2004-09-14 01:33:32 +00:00
|
|
|
|
2009-08-14 14:38:13 +02:00
|
|
|
#define CLI_OFF_ANY 0xffffffff
|
2018-12-03 12:37:58 -05:00
|
|
|
#define CLI_OFF_NONE 0xfffffffe
|
2009-08-14 14:38:13 +02:00
|
|
|
#define CLI_OFF_ABSOLUTE 1
|
|
|
|
|
#define CLI_OFF_EOF_MINUS 2
|
|
|
|
|
#define CLI_OFF_EP_PLUS 3
|
|
|
|
|
#define CLI_OFF_EP_MINUS 4
|
|
|
|
|
#define CLI_OFF_SL_PLUS 5
|
|
|
|
|
#define CLI_OFF_SX_PLUS 6
|
2010-01-04 14:56:04 +01:00
|
|
|
#define CLI_OFF_VERSION 7
|
2010-02-08 13:45:03 +02:00
|
|
|
#define CLI_OFF_MACRO 8
|
2018-12-03 12:37:58 -05:00
|
|
|
#define CLI_OFF_SE 9
|
|
|
|
|
|
|
|
|
|
// clang-format on
|
2009-08-14 14:38:13 +02:00
|
|
|
|
2020-03-19 21:23:54 -04:00
|
|
|
/**
|
|
|
|
|
* @brief Non-magic scan matching using a file buffer for input. Older API
|
|
|
|
|
*
|
|
|
|
|
* This function is lower-level, requiring a call to `cli_exp_eval()` after the
|
|
|
|
|
* match to evaluate logical signatures and yara rules.
|
|
|
|
|
*
|
|
|
|
|
* This function does not perform file type magic identification and does not use
|
|
|
|
|
* the file format scanners.
|
|
|
|
|
*
|
2021-07-14 15:33:52 -07:00
|
|
|
* @param buffer The buffer to be matched.
|
|
|
|
|
* @param length The length of the buffer or amount of bytets to match.
|
|
|
|
|
* @param offset Offset into the buffer from which to start matching.
|
|
|
|
|
* @param ctx The scanning context.
|
|
|
|
|
* @param ftype If specified, may limit signature matching trie by target type corresponding with the specified CL_TYPE
|
|
|
|
|
* @param[in,out] acdata A list of pattern maching data structs to contain match results, one for each pattern matching trie.
|
2020-03-19 21:23:54 -04:00
|
|
|
* @return cl_error_t
|
|
|
|
|
*/
|
2020-03-21 14:15:28 -04:00
|
|
|
cl_error_t cli_scan_buff(const unsigned char *buffer, uint32_t length, uint32_t offset, cli_ctx *ctx, cli_file_t ftype, struct cli_ac_data **acdata);
|
2005-09-23 02:23:36 +00:00
|
|
|
|
2020-03-19 21:23:54 -04:00
|
|
|
/**
|
|
|
|
|
* @brief Non-magic scan matching using a file descriptor for input.
|
|
|
|
|
*
|
|
|
|
|
* This function does not perform file type magic identification and does not use
|
|
|
|
|
* the file format scanners.
|
|
|
|
|
*
|
2020-03-21 14:15:28 -04:00
|
|
|
* This function uses the newer cli_scan_fmap() scanning API.
|
2020-03-19 21:23:54 -04:00
|
|
|
*
|
2021-07-16 11:47:23 -07:00
|
|
|
* @param desc File descriptor to be used for input
|
|
|
|
|
* @param ctx The scanning context.
|
|
|
|
|
* @param ftype If specified, may limit signature matching trie by target type corresponding with the specified CL_TYPE
|
|
|
|
|
* @param ftonly Boolean indicating if the scan is for file-type detection only.
|
|
|
|
|
* @param[out] ftoffset A list of file type signature matches with their corresponding offsets.
|
|
|
|
|
* @param acmode Use AC_SCAN_VIR and AC_SCAN_FT to set scanning modes.
|
|
|
|
|
* @param[out] acres A list of cli_ac_result AC pattern matching results.
|
|
|
|
|
* @param name (optional) Original name of the file (to set fmap name metadata)
|
2022-03-09 22:26:40 -08:00
|
|
|
* @param attributes Layer attributes for the thing to be scanned.
|
2020-03-19 21:23:54 -04:00
|
|
|
* @return cl_error_t
|
|
|
|
|
*/
|
2022-03-09 22:26:40 -08:00
|
|
|
cl_error_t cli_scan_desc(int desc, cli_ctx *ctx, cli_file_t ftype, uint8_t ftonly, struct cli_matched_type **ftoffset,
|
|
|
|
|
unsigned int acmode, struct cli_ac_result **acres, const char *name, uint32_t attributes);
|
2020-03-19 21:23:54 -04:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* @brief Non-magic scan matching of the current fmap in the scan context. Newer API.
|
|
|
|
|
*
|
|
|
|
|
* This API will invoke cli_exp_eval() for you.
|
|
|
|
|
*
|
2021-07-16 11:47:23 -07:00
|
|
|
* @param ctx The scanning context.
|
|
|
|
|
* @param ftype If specified, may limit signature matching trie by target type corresponding with the specified CL_TYPE
|
|
|
|
|
* @param ftonly Boolean indicating if the scan is for file-type detection only.
|
|
|
|
|
* @param[out] ftoffset A list of file type signature matches with their corresponding offsets.
|
|
|
|
|
* @param acmode Use AC_SCAN_VIR and AC_SCAN_FT to set scanning modes.
|
|
|
|
|
* @param[out] acres A list of cli_ac_result AC pattern matching results.
|
|
|
|
|
* @param refhash MD5 hash of the current file, used to save time creating hashes and to limit scan recursion for the HandlerType logical signature FTM feature.
|
2020-03-19 21:23:54 -04:00
|
|
|
* @return cl_error_t
|
|
|
|
|
*/
|
2020-03-21 14:15:28 -04:00
|
|
|
cl_error_t cli_scan_fmap(cli_ctx *ctx, cli_file_t ftype, uint8_t ftonly, struct cli_matched_type **ftoffset, unsigned int acmode, struct cli_ac_result **acres, unsigned char *refhash);
|
2020-03-19 21:23:54 -04:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* @brief Evaluate logical signatures and yara rules given the AC matching results
|
2020-03-21 14:15:28 -04:00
|
|
|
* from cli_scan_buff() / matcher_run().
|
2020-03-19 21:23:54 -04:00
|
|
|
*
|
|
|
|
|
* @param ctx The scanning context.
|
|
|
|
|
* @param root The AC trie root to match with.
|
|
|
|
|
* @param acdata AC match results for a specific AC trie.
|
|
|
|
|
* @param target_info File metadata used to evaluate logical sig and yara rule options.
|
|
|
|
|
* @param hash Reference hash of the current file, used to limit recursion for the HandlerType logical signature FTM feature.
|
|
|
|
|
* @return cl_error_t
|
|
|
|
|
*/
|
2019-02-27 00:47:38 -05:00
|
|
|
cl_error_t cli_exp_eval(cli_ctx *ctx, struct cli_matcher *root, struct cli_ac_data *acdata, struct cli_target_info *target_info, const char *hash);
|
2020-03-19 21:23:54 -04:00
|
|
|
|
2019-02-27 00:47:38 -05:00
|
|
|
cl_error_t cli_caloff(const char *offstr, const struct cli_target_info *info, unsigned int target, uint32_t *offdata, uint32_t *offset_min, uint32_t *offset_max);
|
2006-12-02 00:09:02 +00:00
|
|
|
|
libclamav: Fix scan recursion tracking
Scan recursion is the process of identifying files embedded in other
files and then scanning them, recursively.
Internally this process is more complex than it may sound because a file
may have multiple layers of types before finding a new "file".
At present we treat the recursion count in the scanning context as an
index into both our fmap list AND our container list. These two lists
are conceptually a part of the same thing and should be unified.
But what's concerning is that the "recursion level" isn't actually
incremented or decremented at the same time that we add a layer to the
fmap or container lists but instead is more touchy-feely, increasing
when we find a new "file".
To account for this shadiness, the size of the fmap and container lists
has always been a little longer than our "max scan recursion" limit so
we don't accidentally overflow the fmap or container arrays (!).
I've implemented a single recursion-stack as an array, similar to before,
which includes a pointer to each fmap at each layer, along with the size
and type. Push and pop functions add and remove layers whenever a new
fmap is added. A boolean argument when pushing indicates if the new layer
represents a new buffer or new file (descriptor). A new buffer will reset
the "nested fmap level" (described below).
This commit also provides a solution for an issue where we detect
embedded files more than once during scan recursion.
For illustration, imagine a tarball named foo.tar.gz with this structure:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| baz.exe | PE | 0 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| │ └── hello.txt | ASCII | 2 | 0 |
| └── sfx.7z | 7Z | 1 | 1 |
| └── world.txt | ASCII | 2 | 0 |
(A) If we scan for embedded files at any layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| ├── foo.tar | TAR | 1 | 0 |
| │ ├── bar.zip | ZIP | 2 | 1 |
| │ │ └── hola.txt | ASCII | 3 | 0 |
| │ ├── baz.exe | PE | 2 | 1 |
| │ │ ├── sfx.zip | ZIP | 3 | 1 |
| │ │ │ └── hello.txt | ASCII | 4 | 0 |
| │ │ └── sfx.7z | 7Z | 3 | 1 |
| │ │ └── world.txt | ASCII | 4 | 0 |
| │ ├── sfx.zip | ZIP | 2 | 1 |
| │ │ └── hello.txt | ASCII | 3 | 0 |
| │ └── sfx.7z | 7Z | 2 | 1 |
| │ └── world.txt | ASCII | 3 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| └── sfx.7z | 7Z | 1 | 1 |
(A) is bad because it scans content more than once.
Note that for the GZ layer, it may detect the ZIP and 7Z if the
signature hits on the compressed data, which it might, though
extracting the ZIP and 7Z will likely fail.
The reason the above doesn't happen now is that we restrict embedded
type scans for a bunch of archive formats to include GZ and TAR.
(B) If we scan for embedded files at the foo.tar layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| ├── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 2 | 1 |
| │ └── hello.txt | ASCII | 3 | 0 |
| └── sfx.7z | 7Z | 2 | 1 |
| └── world.txt | ASCII | 3 | 0 |
(B) is almost right. But we can achieve it easily enough only scanning for
embedded content in the current fmap when the "nested fmap level" is 0.
The upside is that it should safely detect all embedded content, even if
it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe.
The biggest risk I can think of affects ZIPs. SFXZIP detection
is identical to ZIP detection, which is why we don't allow SFXZIP to be
detected if insize of a ZIP. If we only allow embedded type scanning at
fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP
if the bar.exe was not compressed in foo.zip and if non-compressed files
extracted from ZIPs aren't extracted as new buffers:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.zip | ZIP | 0 | 0 |
| └── bar.exe | PE | 1 | 1 |
| └── sfx.zip | ZIP | 2 | 2 |
Provided that we ensure all files extracted from zips are scanned in
new buffers, option (B) should be safe.
(C) If we scan for embedded files at the baz.exe layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 3 | 1 |
| │ └── hello.txt | ASCII | 4 | 0 |
| └── sfx.7z | 7Z | 3 | 1 |
| └── world.txt | ASCII | 4 | 0 |
(C) is right. But it's harder to achieve. For this example we can get it by
restricting 7ZSFX and ZIPSFX detection only when scanning an executable.
But that may mean losing detection of archives embedded elsewhere.
And we'd have to identify allowable container types for each possible
embedded type, which would be very difficult.
So this commit aims to solve the issue the (B)-way.
Note that in all situations, we still have to scan with file typing
enabled to determine if we need to reassign the current file type, such
as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2-
compressed. Detection of DMG and a handful of other types rely on
finding data partway through or near the ned of a file before
reassigning the entire file as the new type.
Other fixes and considerations in this commit:
- The utf16 HTML parser has weak error handling, particularly with respect
to creating a nested fmap for scanning the ascii decoded file.
This commit cleans up the error handling and wraps the nested scan with
the recursion-stack push()/pop() for correct recursion tracking.
Before this commit, each container layer had a flag to indicate if the
container layer is valid.
We need something similar so that the cli_recursion_stack_get_*()
functions ignore normalized layers. Details...
Imagine an LDB signature for HTML content that specifies a ZIP
container. If the signature actually alerts on the normalized HTML and
you don't ignore normalized layers for the container check, it will
appear as though the alert is in an HTML container rather than a ZIP
container.
This commit accomplishes this with a boolean you set in the scan context
before scanning a new layer. Then when the new fmap is created, it will
use that flag to set similar flag for the layer. The context flag is
reset those that anything after this doesn't have that flag.
The flag allows the new recursion_stack_get() function to ignore
normalized layers when iterating the stack to return a layer at a
requested index, negative or positive.
Scanning normalized extracted/normalized javascript and VBA should also
use the 'layer is normalized' flag.
- This commit also fixes Heuristic.Broken.Executable alert for ELF files
to make sure that:
A) these only alert if cli_append_virus() returns CL_VIRUS (aka it
respects the FP check).
B) all broken-executable alerts for ELF only happen if the
SCAN_HEURISTIC_BROKEN option is enabled.
- This commit also cleans up the error handling in cli_magic_scan_dir().
This was needed so we could correctly apply the layer-is-normalized-flag
to all VBA macros extracted to a directory when scanning the directory.
- Also fix an issue where exceeding scan maximums wouldn't cause embedded
file detection scans to abort. Granted we don't actually want to abort
if max filesize or max recursion depth are exceeded... only if max
scansize, max files, and max scantime are exceeded.
Add 'abort_scan' flag to scan context, to protect against depending on
correct error propagation for fatal conditions. Instead, setting this
flag in the scan context should guarantee that a fatal condition deep in
scan recursion isn't lost which result in more stuff being scanned
instead of aborting. This shouldn't be necessary, but some status codes
like CL_ETIMEOUT never used to be fatal and it's easier to do this than
to verify every parser only returns CL_ETIMEOUT and other "fatal
status codes" in fatal conditions.
- Remove duplicate is_tar() prototype from filestypes.c and include
is_tar.h instead.
- Presently we create the fmap hash when creating the fmap.
This wastes a bit of CPU if the hash is never needed.
Now that we're creating fmap's for all embedded files discovered with
file type recognition scans, this is a much more frequent occurence and
really slows things down.
This commit fixes the issue by only creating fmap hashes as needed.
This should not only resolve the perfomance impact of creating fmap's
for all embedded files, but also should improve performance in general.
- Add allmatch check to the zip parser after the central-header meta
match. That way we don't multiple alerts with the same match except in
allmatch mode. Clean up error handling in the zip parser a tiny bit.
- Fixes to ensure that the scan limits such as scansize, filesize,
recursion depth, # of embedded files, and scantime are always reported
if AlertExceedsMax (--alert-exceeds-max) is enabled.
- Fixed an issue where non-fatal alerts for exceeding scan maximums may
mask signature matches later on. I changed it so these alerts use the
"possibly unwanted" alert-type and thus only alert if no other alerts
were found or if all-match or heuristic-precedence are enabled.
- Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata
when the --gen-json feature is enabled. These will show up once under
"ParseErrors" the first time a limit is exceeded. In the present
implementation, only one limits-exceeded events will be added, so as to
prevent a malicious or malformed sample from filling the JSON buffer
with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
|
|
|
/**
|
|
|
|
|
* @brief Determine if an alert is a known false positive, using each fmap in the the ctx->container stack to check MD5, SHA1, and SHA256 hashes.
|
|
|
|
|
*
|
|
|
|
|
* @param ctx The scanning context.
|
|
|
|
|
* @param vname (Optional) The name of the signature alert.
|
|
|
|
|
* @return cl_error_t CL_CLEAN If an allow-list hash matches with one of the fmap hashes in the scan recursion stack.
|
|
|
|
|
* CL_VIRUS If no allow-list hash matches.
|
|
|
|
|
*/
|
|
|
|
|
cl_error_t cli_check_fp(cli_ctx *ctx, const char *vname);
|
2009-04-23 13:24:21 +00:00
|
|
|
|
2019-02-27 00:47:38 -05:00
|
|
|
cl_error_t cli_matchmeta(cli_ctx *ctx, const char *fname, size_t fsizec, size_t fsizer, int encrypted, unsigned int filepos, int res1, void *res2);
|
2010-01-07 18:26:12 +01:00
|
|
|
|
libclamav: Fix scan recursion tracking
Scan recursion is the process of identifying files embedded in other
files and then scanning them, recursively.
Internally this process is more complex than it may sound because a file
may have multiple layers of types before finding a new "file".
At present we treat the recursion count in the scanning context as an
index into both our fmap list AND our container list. These two lists
are conceptually a part of the same thing and should be unified.
But what's concerning is that the "recursion level" isn't actually
incremented or decremented at the same time that we add a layer to the
fmap or container lists but instead is more touchy-feely, increasing
when we find a new "file".
To account for this shadiness, the size of the fmap and container lists
has always been a little longer than our "max scan recursion" limit so
we don't accidentally overflow the fmap or container arrays (!).
I've implemented a single recursion-stack as an array, similar to before,
which includes a pointer to each fmap at each layer, along with the size
and type. Push and pop functions add and remove layers whenever a new
fmap is added. A boolean argument when pushing indicates if the new layer
represents a new buffer or new file (descriptor). A new buffer will reset
the "nested fmap level" (described below).
This commit also provides a solution for an issue where we detect
embedded files more than once during scan recursion.
For illustration, imagine a tarball named foo.tar.gz with this structure:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
But suppose baz.exe embeds a ZIP archive and a 7Z archive, like this:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| baz.exe | PE | 0 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| │ └── hello.txt | ASCII | 2 | 0 |
| └── sfx.7z | 7Z | 1 | 1 |
| └── world.txt | ASCII | 2 | 0 |
(A) If we scan for embedded files at any layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| ├── foo.tar | TAR | 1 | 0 |
| │ ├── bar.zip | ZIP | 2 | 1 |
| │ │ └── hola.txt | ASCII | 3 | 0 |
| │ ├── baz.exe | PE | 2 | 1 |
| │ │ ├── sfx.zip | ZIP | 3 | 1 |
| │ │ │ └── hello.txt | ASCII | 4 | 0 |
| │ │ └── sfx.7z | 7Z | 3 | 1 |
| │ │ └── world.txt | ASCII | 4 | 0 |
| │ ├── sfx.zip | ZIP | 2 | 1 |
| │ │ └── hello.txt | ASCII | 3 | 0 |
| │ └── sfx.7z | 7Z | 2 | 1 |
| │ └── world.txt | ASCII | 3 | 0 |
| ├── sfx.zip | ZIP | 1 | 1 |
| └── sfx.7z | 7Z | 1 | 1 |
(A) is bad because it scans content more than once.
Note that for the GZ layer, it may detect the ZIP and 7Z if the
signature hits on the compressed data, which it might, though
extracting the ZIP and 7Z will likely fail.
The reason the above doesn't happen now is that we restrict embedded
type scans for a bunch of archive formats to include GZ and TAR.
(B) If we scan for embedded files at the foo.tar layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| ├── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 2 | 1 |
| │ └── hello.txt | ASCII | 3 | 0 |
| └── sfx.7z | 7Z | 2 | 1 |
| └── world.txt | ASCII | 3 | 0 |
(B) is almost right. But we can achieve it easily enough only scanning for
embedded content in the current fmap when the "nested fmap level" is 0.
The upside is that it should safely detect all embedded content, even if
it may think the sfz.zip and sfx.7z are in foo.tar instead of in baz.exe.
The biggest risk I can think of affects ZIPs. SFXZIP detection
is identical to ZIP detection, which is why we don't allow SFXZIP to be
detected if insize of a ZIP. If we only allow embedded type scanning at
fmap-layer 0 in each buffer, this will fail to detect the embedded ZIP
if the bar.exe was not compressed in foo.zip and if non-compressed files
extracted from ZIPs aren't extracted as new buffers:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.zip | ZIP | 0 | 0 |
| └── bar.exe | PE | 1 | 1 |
| └── sfx.zip | ZIP | 2 | 2 |
Provided that we ensure all files extracted from zips are scanned in
new buffers, option (B) should be safe.
(C) If we scan for embedded files at the baz.exe layer, we may detect:
| description | type | rec level | nested fmap level |
| ------------------------- | ----- | --------- | ----------------- |
| foo.tar.gz | GZ | 0 | 0 |
| └── foo.tar | TAR | 1 | 0 |
| ├── bar.zip | ZIP | 2 | 1 |
| │ └── hola.txt | ASCII | 3 | 0 |
| └── baz.exe | PE | 2 | 1 |
| ├── sfx.zip | ZIP | 3 | 1 |
| │ └── hello.txt | ASCII | 4 | 0 |
| └── sfx.7z | 7Z | 3 | 1 |
| └── world.txt | ASCII | 4 | 0 |
(C) is right. But it's harder to achieve. For this example we can get it by
restricting 7ZSFX and ZIPSFX detection only when scanning an executable.
But that may mean losing detection of archives embedded elsewhere.
And we'd have to identify allowable container types for each possible
embedded type, which would be very difficult.
So this commit aims to solve the issue the (B)-way.
Note that in all situations, we still have to scan with file typing
enabled to determine if we need to reassign the current file type, such
as re-identifying a Bzip2 archive as a DMG that happens to be Bzip2-
compressed. Detection of DMG and a handful of other types rely on
finding data partway through or near the ned of a file before
reassigning the entire file as the new type.
Other fixes and considerations in this commit:
- The utf16 HTML parser has weak error handling, particularly with respect
to creating a nested fmap for scanning the ascii decoded file.
This commit cleans up the error handling and wraps the nested scan with
the recursion-stack push()/pop() for correct recursion tracking.
Before this commit, each container layer had a flag to indicate if the
container layer is valid.
We need something similar so that the cli_recursion_stack_get_*()
functions ignore normalized layers. Details...
Imagine an LDB signature for HTML content that specifies a ZIP
container. If the signature actually alerts on the normalized HTML and
you don't ignore normalized layers for the container check, it will
appear as though the alert is in an HTML container rather than a ZIP
container.
This commit accomplishes this with a boolean you set in the scan context
before scanning a new layer. Then when the new fmap is created, it will
use that flag to set similar flag for the layer. The context flag is
reset those that anything after this doesn't have that flag.
The flag allows the new recursion_stack_get() function to ignore
normalized layers when iterating the stack to return a layer at a
requested index, negative or positive.
Scanning normalized extracted/normalized javascript and VBA should also
use the 'layer is normalized' flag.
- This commit also fixes Heuristic.Broken.Executable alert for ELF files
to make sure that:
A) these only alert if cli_append_virus() returns CL_VIRUS (aka it
respects the FP check).
B) all broken-executable alerts for ELF only happen if the
SCAN_HEURISTIC_BROKEN option is enabled.
- This commit also cleans up the error handling in cli_magic_scan_dir().
This was needed so we could correctly apply the layer-is-normalized-flag
to all VBA macros extracted to a directory when scanning the directory.
- Also fix an issue where exceeding scan maximums wouldn't cause embedded
file detection scans to abort. Granted we don't actually want to abort
if max filesize or max recursion depth are exceeded... only if max
scansize, max files, and max scantime are exceeded.
Add 'abort_scan' flag to scan context, to protect against depending on
correct error propagation for fatal conditions. Instead, setting this
flag in the scan context should guarantee that a fatal condition deep in
scan recursion isn't lost which result in more stuff being scanned
instead of aborting. This shouldn't be necessary, but some status codes
like CL_ETIMEOUT never used to be fatal and it's easier to do this than
to verify every parser only returns CL_ETIMEOUT and other "fatal
status codes" in fatal conditions.
- Remove duplicate is_tar() prototype from filestypes.c and include
is_tar.h instead.
- Presently we create the fmap hash when creating the fmap.
This wastes a bit of CPU if the hash is never needed.
Now that we're creating fmap's for all embedded files discovered with
file type recognition scans, this is a much more frequent occurence and
really slows things down.
This commit fixes the issue by only creating fmap hashes as needed.
This should not only resolve the perfomance impact of creating fmap's
for all embedded files, but also should improve performance in general.
- Add allmatch check to the zip parser after the central-header meta
match. That way we don't multiple alerts with the same match except in
allmatch mode. Clean up error handling in the zip parser a tiny bit.
- Fixes to ensure that the scan limits such as scansize, filesize,
recursion depth, # of embedded files, and scantime are always reported
if AlertExceedsMax (--alert-exceeds-max) is enabled.
- Fixed an issue where non-fatal alerts for exceeding scan maximums may
mask signature matches later on. I changed it so these alerts use the
"possibly unwanted" alert-type and thus only alert if no other alerts
were found or if all-match or heuristic-precedence are enabled.
- Added the "Heuristics.Limits.Exceeded.*" events to the JSON metadata
when the --gen-json feature is enabled. These will show up once under
"ParseErrors" the first time a limit is exceeded. In the present
implementation, only one limits-exceeded events will be added, so as to
prevent a malicious or malformed sample from filling the JSON buffer
with millions of events and using a tonne of RAM.
2021-09-11 14:15:21 -07:00
|
|
|
/** Parse the executable headers and, if successful, populate exeinfo
|
|
|
|
|
*
|
|
|
|
|
* If target refers to a supported executable file type, the exe header
|
|
|
|
|
* will be parsed and, if successful, info->status will be set to 1.
|
|
|
|
|
* If parsing the exe header fails, info->status will be set to -1.
|
|
|
|
|
* The caller MUST destroy info via a call to cli_targetinfo_destroy
|
|
|
|
|
* regardless of what info->status is set to.
|
|
|
|
|
*
|
|
|
|
|
* @param info A structure to populate with info from the exe header. This
|
|
|
|
|
* MUST be initialized via cli_targetinfo_init prior to calling
|
|
|
|
|
* @param target the target executable file type. Possible values are:
|
|
|
|
|
* - 1 - PE32 / PE32+
|
|
|
|
|
* - 6 - ELF
|
|
|
|
|
* - 9 - MachO
|
|
|
|
|
* @param ctx The current scan context
|
|
|
|
|
*/
|
|
|
|
|
void cli_targetinfo(struct cli_target_info *info, unsigned int target, cli_ctx *ctx);
|
2013-12-05 15:09:19 -08:00
|
|
|
|
2003-07-29 15:48:06 +00:00
|
|
|
#endif
|