Change the clean-cache to use SHA2-256 instead of MD5.
Note that all references are changed to specify "SHA2-256" now instead
of "SHA256", for clarity. But there is no plan to add support for SHA3
algorithms at this time.
Significant code cleanup. E.g.:
- Implemented goto-done error handling.
- Used `uint8_t *` instead of `unsigned char *`.
- Use `bool` for boolean checks, rather than `int.
- Used `#defines` instead of magic numbers.
- Removed duplicate `#defines` for things like hash length.
Add new option to calculate and record additional hash types when the
"generate metadata JSON" feature is enabled:
- libclamav option: `CL_SCAN_GENERAL_STORE_EXTRA_HASHES`
- clamscan option: `--json-store-extra-hashes` (default off)
- clamd.conf option: `JsonStoreExtraHashes` (default 'no')
Renamed the sigtool option `--sha256` to `--sha2-256`.
The original option is still functional, but is deprecated.
For the "generate metadata JSON" feature, the file hash is now stored as
"sha2-256" instead of "FileMD5". If you enable the "extra hashes" option,
then it will also record "md5" and "sha1".
Deprecate and disable the internal "SHA collect" feature.
This option had been hidden behind C #ifdef checks for an option that
wasn't exposed through CMake, so it was basically unavailable anyways.
Changes to calculate file hashes when they're needed and no sooner.
For the FP feature in the matcher module, I have mimiced the
optimization in the FMAP scan routine which makes it so that it can
calculate multiple hashes in a single pass of the file.
The `HandlerType` feature stores a hash of the file in the scan ctx to
prevent retyping the exact same data more than once.
I removed that hash field and replaced it with an attribute flag that is
applied to the new recursion stack layer when retyping a file.
This also closes a minor bug that would prevent retyping a file with an
all-zero hash. :)
The work upgrading cache.c to support SHA2-256 sized hashes thanks to:
https://github.com/m-sola
CLAM-255
CLAM-1858
CLAM-1859
CLAM-1860
* Added loglevel parameter to logg()
* Fix logg and mprintf internals with new loglevels
* Update all logg calls to set loglevel
* Update all mprintf calls to set loglevel
* Fix hidden logg calls
* Executed clam-format
The named "shared" is confusing, especially now that these features are
built as a static library instead of being directly compiled into the
various applications.
If you set an ExcludePath regex in clamd.conf and then perform a
ClamDScan scan with --fdpass --multiscan, it will segfault.
The same issue also affects --fdpass --multiscan scans when using
ExcludePath when scanning a patch that doesn't exist.
The issue is that the filepath isn't being passed along for the path
exclusion regex match, resulting in a NULL deref.
This commit also fixes a possible memory leak if by duplicating the path
for the handle_entry() call _after_ the callback() runs, in case ret
isn't CL_SUCCESS and the function exits without every using the entry
structure or free'ing the copied filename.
The above work temporarily caused a test failure in check_clamd and a
valgrind failure in clamd for the nonexistent file test due to a minor
memory leak. This made it apparent that there were a few other nearby
possible memory leaks.
This commit fixes the above plus cleans up the error handling in clamd's
the file tree walk functions.
The clamd TOCTOU access check fix introduced and expectation that the
scanfile API will set errno if access was denied. We should instead use
the cl_error_t error code enum.
Also added Duane Waddle to the 0.104 contributors acknowledgements.
Clamd presently uses an access() check prior to a scan to see if the
process has permission to open the file. This is problematic for two
reasons.
1. There's a time-of-check/time-of-use issue with checking access
before accessing because the permissions could change in between.
2. The POSIX CAP_DAC_READ_SEARCH feature won't work with access().
CAP_DAC_READ_SEARCH is a posix capability that allows the kernel to
bless a process, saying "You can read any file just like you were root."
The service manager can run clamd and bless it at run time with
CAP_DAC_READ_SEARCH such that fd-passing is not required for clamd to
read files it would otherwise not have permission to access.
This improvement looks up the filename given the file descriptor.
This is supported on Mac and Linux but not presently supported
on other UNIX operating systems. FD-passing is not available on
Windows.
On supported systems, the verdict in the clamd log and the VirusEvent
will show the actual file path instead of something like fd[14].
Users have complained about two specific log events that are extremely
verbose in non-critical error conditions:
- clamonacc reports "ERROR: Can't send to clamd: Bad address"
This may occur when small files are created/destroyed before they can
be sent to be scanned. The log message probably should only be
reported in verbose mode.
- clamonacc reports "ClamMisc: $/proc/XXX vanished before UIDs could be
excluded; scanning anyway"
This may occur when a process that accessed a file exits before
clamonacc find out who accessed the file. This is a fairly frequent
occurence. It can still be problematic if `clamd` was the process which
accessed the file (like a clamd temp file if watching /tmp), generally
it's not an issue and we want to silently scan it anyways.
Also addressed copypaste issue in onas_send_stream() wherein fd is set
to 0 (aka STDIN) if the provided fd == 0 (should've been -1 for invalid
FD) and if filename == NULL. In fact clamonacc never scans STDIN so the
scan should fail if filename == NULL and the provided FD is invalid
(-1).
I also found that "Access denied. ERROR" is easily provoked when using
--fdpass or --stream using this simple script:
for i in {1..5000}; do echo "blah $i" > tmp-$i && rm tmp-$i; done
Clamdscan does not allow for scans to fail quietly because the file does
not exist, but for clamonacc it's a common thing and we don't want to
output an error. To solve this, I changed it so a return length of -1
will still result in an "internal error" message but return len 0
failures will be silently ignored.
I've added a static variable to onas_client_scan() that keeps state in
case clamd is stopped and started - that way it won't print an error
message for every event when offline. Instead it will log an error for
the first connection failure, and log again when the connection is
re-established for a future scan. Calls to onas_client_scan() are
already wrapped with the onas_scan_lock mutex so the static variable
should be safe.
Finally, there were a couple of error responses from clamd that can
occur if the file isn't found which we want to silently ignore, so I've
tweaked the code which checks for specific error messages to account for
these.
The security improvement to perform file realpath lookups prior to a
scan has the adverse effect of causing file scans to fail on Windows
when scanning on some filesystems.
Specifically, it was observed that the ImDisk driver doesn't handle the
IRP_MJ_QUERY_INFORMATION message so the call to look up the realpath
using GetFinalPathNameByHandleW() doesn't work.
There are two other API's I've found which can query the real file path.
The first is to create a file mapping of the target file and then use
GetMappedFileNameW() to get the file path. The other is to use the
NtQueryObject() undocumented NT API to get the file path. Each of
these should return roughly the same thing. For files in an ImDisk
RAM-disk drive, the resulting filepath for R:\clam.exe would
be \\Device\ImDisk0\clam.exe. The trouble is, mapping
\\Device\ImDisk0\clam.exe back to R:\clam.exe would rely on an
assumption that ImDisk is using the default drive letter, which is a tad
hacky.
Instead, this patch simply allows the scan to proceed if the realpath
lookup failed. If the user is using the quarantine (remove/move)
features AND if the scan target filepath has a directory junction (soft
link), then the quarantine action will fail. It's not ideal but it is
quite unlikely.
This patch adds experimental-quality CMake build tooling.
The libmspack build required a modification to use "" instead of <> for
header #includes. This will hopefully be included in the libmspack
upstream project when adding CMake build tooling to libmspack.
Removed use of libltdl when using CMake.
Flex & Bison are now required to build.
If -DMAINTAINER_MODE, then GPERF is also required, though it currently
doesn't actually do anything. TODO!
I found that the autotools build system was generating the lexer output
but not actually compiling it, instead using previously generated (and
manually renamed) lexer c source. As a consequence, changes to the .l
and .y files weren't making it into the build. To resolve this, I
removed generated flex/bison files and fixed the tooling to use the
freshly generated files. Flex and bison are now required build tools.
On Windows, this adds a dependency on the winflexbison package,
which can be obtained using Chocolatey or may be manually installed.
CMake tooling only has partial support for building with external LLVM
library, and no support for the internal LLVM (to be removed in the
future). I.e. The CMake build currently only supports the bytecode
interpreter.
Many files used include paths relative to the top source directory or
relative to the current project, rather than relative to each build
target. Modern CMake support requires including internal dependency
headers the same way you would external dependency headers (albeit
with "" instead of <>). This meant correcting all header includes to
be relative to the build targets and not relative to the workspace.
For example, ...
```c
include "../libclamav/clamav.h"
include "clamd/clamd_others.h"
```
... becomes:
```c
// libclamav
include "clamav.h"
// clamd
include "clamd_others.h"
```
Fixes header name conflicts by renaming a few of the files.
Converted the "shared" code into a static library, which depends on
libclamav. The ironically named "shared" static library provides
features common to the ClamAV apps which are not required in
libclamav itself and are not intended for use by downstream projects.
This change was required for correct modern CMake practices but was
also required to use the automake "subdir-objects" option.
This eliminates warnings when running autoreconf which, in the next
version of autoconf & automake are likely to break the build.
libclamav used to build in multiple stages where an earlier stage is
a static library containing utils required by the "shared" code.
Linking clamdscan and clamdtop with this libclamav utils static lib
allowed these two apps to function without libclamav. While this is
nice in theory, the practical gains are minimal and it complicates
the build system. As such, the autotools and CMake tooling was
simplified for improved maintainability and this feature was thrown
out. clamdtop and clamdscan now require libclamav to function.
Removed the nopthreads version of the autotools
libclamav_internal_utils static library and added pthread linking to
a couple apps that may have issues building on some platforms without
it, with the intention of removing needless complexity from the
source. Kept the regular version of libclamav_internal_utils.la
though it is no longer used anywhere but in libclamav.
Added an experimental doxygen build option which attempts to build
clamav.h and libfreshclam doxygen html docs.
The CMake build tooling also may build the example program(s), which
isn't a feature in the Autotools build system.
Changed C standard to C90+ due to inline linking issues with socket.h
when linking libfreshclam.so on Linux.
Generate common.rc for win32.
Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef
from misc.c.
Add CMake files to the automake dist, so users can try the new
CMake tooling w/out having to build from a git clone.
clamonacc changes:
- Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other
similar macros.
- Added a new clamav-clamonacc.service systemd unit file, based on
the work of ChadDevOps & Aaron Brighton.
- Added missing clamonacc man page.
Updates to clamdscan man page, add missing options.
Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use
libclamav).
Rename Windows mspack.dll to libmspack.dll so all ClamAV-built
libraries have the lib-prefix with Visual Studio as with CMake.
This patch relocates the real-path check from clamdscan and clamonacc
to clamd. While clamonacc is unlikely to send directories or symlinks
to be scanned, clamdscan may send directories. Real-path checks have
to be performed on the files, not the directories -- both because the
directories may contain symlinks and because the cli_realpath()
function wasn't written to support directories on Windows.
Removing problematic call to convert file descriptors to filepaths.
Added filename and tempfile names to scandesc calls in clamd.
Added a general scan option to treat the scan engine as unprivileged,
meaning that the scan engine will not have read access to the file.
Added check to drop a temp file for RAR's where the we don't have
read access to the filepath provided (i.e. unprivileged is set, or
access() check fails).
Updated libclamav documentation detailing new scan options structure.
Renamed references to 'algorithmic' detection to 'heuristic' detection. Renaming references to 'properties' to 'collect metadata'.
Renamed references to 'scan all' to 'scan all match'.
Renamed a couple of 'Hueristic.*' signature names as 'Heuristics.*' signatures (plural) to match majority of other heuristics.
To enable prelude compile with ./configure --enable-prelude
In ClamAV configuration file set Prelude Enable on yes and choose an analyzer name with PreludeAnalyzerName (default ClamAV).
You need to have a prelude manager to use this module.
