There are a large number of allocations for fix sized buffers using the
`cli_malloc` and `cli_calloc` calls that check if the requested size is
larger than our allocation threshold for allocations based on untrusted
input. These allocations will *always* be higher than the threshold, so
the extra stack frame and check for these calls is a waste of CPU.
This commit replaces needless calls with A -> B:
- cli_malloc -> malloc
- cli_calloc -> calloc
- CLI_MALLOC -> MALLOC
- CLI_CALLOC -> CALLOC
I also noticed that our MPOOL_MALLOC / MPOOL_CALLOC are not limited by
the max-allocation threshold, when MMAP is found/enabled. But the
alternative was set to cli_malloc / cli_calloc when disabled. I changed
those as well.
I didn't change the cli_realloc/2 calls because our version of realloc
not only implements a threshold but also stabilizes the undefined
behavior in realloc to protect against accidental double-free's.
It may be worth implementing a cli_realloc that doesn't have the
threshold built-in, however, so as to allow reallocaitons for things
like buffers for loading signatures, which aren't subject to the same
concern as allocations for scanning possible malware.
There was one case in mbox.c where I changed MALLOC -> CLI_MALLOC,
because it appears to be allocating based on untrusted input.
Prototypes (or the declarations themselves, if there is no
corresponding prototype) for functions that take no arguments are
required by the C standard to specify (void) as their argument list;
for example,
regex_pcre.h:79:1: error: function declaration isn't a prototype
[-Werror=strict-prototypes]
79 | cl_error_t cli_pcre_init_internal();
Future versions of clang may become strict about this, and there's no
harm in conforming to the standard right now, so we fix all such
instances in this commit.
A race condition existed where clamonacc would call logg and attempt to
write to a logfile either before the logg interface had been initialized
or after it had been cleaned up.
This happens due to logg calls at cleanup during asynchronous thread
shutdowns, and during startup when watching directories with ongoing
event triggers.
This resulted in new files with garbage-filled names being created and
written to under the initial process' runtime path.
Changes:
Move logg setup to start of clamonacc.c main()
Change all raceable calls to logg to mprintf instead
* Added loglevel parameter to logg()
* Fix logg and mprintf internals with new loglevels
* Update all logg calls to set loglevel
* Update all mprintf calls to set loglevel
* Fix hidden logg calls
* Executed clam-format
The named "shared" is confusing, especially now that these features are
built as a static library instead of being directly compiled into the
various applications.
Enabled the metadata collection feature, scan heuristics, and all-match
mode when fuzzing in the interest of better code coverage.
Also remove deprecated STREAM command.
I faced a problem with tar application hang in case clamonacc is active.
Tar is extracting a lot of small files from an archive and stops at some
arbitrary point. The problem is not stable to reproduce.
I can explain it in the following way:
1. Consumer thread is waiting for condition variable which indicates
that there are files in the queue. Once the condition is satisfied, the
consumer starts dispatching the files to the worker threads in the
thread pool.
2. While the consumer thread processes the files in the queue, producer
can put few more items in the queue and fire the condition variable.
3. Consumer thread stucks on waiting for condition variable
nevertheless there are items in the queue. However, new items are not
coming from tar application because it is waiting on verdict regarding
the items in the queue.
The solution is to:
1. Use a single mutex to guard the queue and the condition variable
2. Signal condition variable before releasing mutex in producer thread
3. Check if there are events in the queue before waiting for condition
Besides, some small things worth to mention:
1. pthread_testcancel() call is not required as pthread_cond_wait()
does the check for thread cancelation.
2. Lock seems to be not required in onas_scan_queue_exit() as by this
time no one supposed to access the event queue
3. Memory allocation in onas_queue_event() is done without locking the
queue
4. Dispatch an item to a worker thread is done without locking the
queue (calling thpool_add_work())
Also shutdown ddd thread before event processor thread. This should
prevent inserting events to already destroyed queue
clamonacc's --wait option was broken and would exit as soon as clamd
responded, rather than starting clamonacc. The fix is simply to return
"success" when the pong is received, rather than "break".
clamonacc's --watch-list option's short-hand "-w" conflicts with the
--wait option's "-w" short-hand. This causes --watch-list to be
non-functional, invoking the --wait option when you use --watch-list.
This patch switches the --watch-list short-hand to "-W".
Move from using curl when attempting to pass file descriptors to
using system calls
System calls must be used here since the kernel translates file
descriptors from one process to another internally when passed
Default --wait timeout adjusted from 29 to 30 seconds.
--ping and --wait should exit with CL_ETIMEOUT (21) on timeout.
--ping should only return exit code 0 if clamd responds.
Silenced a couple switch fall-through warnings.
Added proc_fd_fname stack buffer to use with readlink, because the
pointers are restricted and using the same buffer with readlink could
result in undefined behavior.
Relocated clamonacc log verbosity initialization so early verbose log
messages will be printed.
Added a new status code for clamonacc startup checks so the --ping
feature can exit the process early with exit code 0.
Update the NEWS to add and correct content prior to the release
candidate.
Changed the version string to have the -rc suffix.
Also fixed a couple of --help and manpage issues.
This patch adds experimental-quality CMake build tooling.
The libmspack build required a modification to use "" instead of <> for
header #includes. This will hopefully be included in the libmspack
upstream project when adding CMake build tooling to libmspack.
Removed use of libltdl when using CMake.
Flex & Bison are now required to build.
If -DMAINTAINER_MODE, then GPERF is also required, though it currently
doesn't actually do anything. TODO!
I found that the autotools build system was generating the lexer output
but not actually compiling it, instead using previously generated (and
manually renamed) lexer c source. As a consequence, changes to the .l
and .y files weren't making it into the build. To resolve this, I
removed generated flex/bison files and fixed the tooling to use the
freshly generated files. Flex and bison are now required build tools.
On Windows, this adds a dependency on the winflexbison package,
which can be obtained using Chocolatey or may be manually installed.
CMake tooling only has partial support for building with external LLVM
library, and no support for the internal LLVM (to be removed in the
future). I.e. The CMake build currently only supports the bytecode
interpreter.
Many files used include paths relative to the top source directory or
relative to the current project, rather than relative to each build
target. Modern CMake support requires including internal dependency
headers the same way you would external dependency headers (albeit
with "" instead of <>). This meant correcting all header includes to
be relative to the build targets and not relative to the workspace.
For example, ...
```c
include "../libclamav/clamav.h"
include "clamd/clamd_others.h"
```
... becomes:
```c
// libclamav
include "clamav.h"
// clamd
include "clamd_others.h"
```
Fixes header name conflicts by renaming a few of the files.
Converted the "shared" code into a static library, which depends on
libclamav. The ironically named "shared" static library provides
features common to the ClamAV apps which are not required in
libclamav itself and are not intended for use by downstream projects.
This change was required for correct modern CMake practices but was
also required to use the automake "subdir-objects" option.
This eliminates warnings when running autoreconf which, in the next
version of autoconf & automake are likely to break the build.
libclamav used to build in multiple stages where an earlier stage is
a static library containing utils required by the "shared" code.
Linking clamdscan and clamdtop with this libclamav utils static lib
allowed these two apps to function without libclamav. While this is
nice in theory, the practical gains are minimal and it complicates
the build system. As such, the autotools and CMake tooling was
simplified for improved maintainability and this feature was thrown
out. clamdtop and clamdscan now require libclamav to function.
Removed the nopthreads version of the autotools
libclamav_internal_utils static library and added pthread linking to
a couple apps that may have issues building on some platforms without
it, with the intention of removing needless complexity from the
source. Kept the regular version of libclamav_internal_utils.la
though it is no longer used anywhere but in libclamav.
Added an experimental doxygen build option which attempts to build
clamav.h and libfreshclam doxygen html docs.
The CMake build tooling also may build the example program(s), which
isn't a feature in the Autotools build system.
Changed C standard to C90+ due to inline linking issues with socket.h
when linking libfreshclam.so on Linux.
Generate common.rc for win32.
Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef
from misc.c.
Add CMake files to the automake dist, so users can try the new
CMake tooling w/out having to build from a git clone.
clamonacc changes:
- Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other
similar macros.
- Added a new clamav-clamonacc.service systemd unit file, based on
the work of ChadDevOps & Aaron Brighton.
- Added missing clamonacc man page.
Updates to clamdscan man page, add missing options.
Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use
libclamav).
Rename Windows mspack.dll to libmspack.dll so all ClamAV-built
libraries have the lib-prefix with Visual Studio as with CMake.
The clamonacc command doesn't present a `--debug` flag, but according
to your blog https://blog.clamav.net/2019/09/understanding-and-transitioning-to.html
the correct flag should be `--verbose`:
"[...]This is akin to clamd’s or clamscan’s --debug option, but isn’t
quite so noisy as either of those. By default, clamonacc does not
print any output after daemonizing, so you will have to pair this
option with --log or --foreground to use it.[...]"
@kolbma
kolbma onas_...cleanup function return void
The functions
onas_cleanup()
onas_context_cleanup()
doesn't return anything so we need type void and not void*.
