clamav/clamonacc/scan/onas_queue.c

284 lines
7.7 KiB
C
Raw Normal View History

/*
2025-02-14 10:24:30 -05:00
* Copyright (C) 2019-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
*
* Authors: Mickey Sola
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#if HAVE_CONFIG_H
#include "clamav-config.h"
#endif
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <signal.h>
#include <pthread.h>
#if defined(__linux__)
#include <sys/prctl.h>
#endif
#include <string.h>
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
// libclamav
#include "clamav.h"
// common
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
#include "optparser.h"
#include "output.h"
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
#include "../misc/utils.h"
#include "../c-thread-pool/thpool.h"
#include "thread.h"
Add CMake build tooling This patch adds experimental-quality CMake build tooling. The libmspack build required a modification to use "" instead of <> for header #includes. This will hopefully be included in the libmspack upstream project when adding CMake build tooling to libmspack. Removed use of libltdl when using CMake. Flex & Bison are now required to build. If -DMAINTAINER_MODE, then GPERF is also required, though it currently doesn't actually do anything. TODO! I found that the autotools build system was generating the lexer output but not actually compiling it, instead using previously generated (and manually renamed) lexer c source. As a consequence, changes to the .l and .y files weren't making it into the build. To resolve this, I removed generated flex/bison files and fixed the tooling to use the freshly generated files. Flex and bison are now required build tools. On Windows, this adds a dependency on the winflexbison package, which can be obtained using Chocolatey or may be manually installed. CMake tooling only has partial support for building with external LLVM library, and no support for the internal LLVM (to be removed in the future). I.e. The CMake build currently only supports the bytecode interpreter. Many files used include paths relative to the top source directory or relative to the current project, rather than relative to each build target. Modern CMake support requires including internal dependency headers the same way you would external dependency headers (albeit with "" instead of <>). This meant correcting all header includes to be relative to the build targets and not relative to the workspace. For example, ... ```c include "../libclamav/clamav.h" include "clamd/clamd_others.h" ``` ... becomes: ```c // libclamav include "clamav.h" // clamd include "clamd_others.h" ``` Fixes header name conflicts by renaming a few of the files. Converted the "shared" code into a static library, which depends on libclamav. The ironically named "shared" static library provides features common to the ClamAV apps which are not required in libclamav itself and are not intended for use by downstream projects. This change was required for correct modern CMake practices but was also required to use the automake "subdir-objects" option. This eliminates warnings when running autoreconf which, in the next version of autoconf & automake are likely to break the build. libclamav used to build in multiple stages where an earlier stage is a static library containing utils required by the "shared" code. Linking clamdscan and clamdtop with this libclamav utils static lib allowed these two apps to function without libclamav. While this is nice in theory, the practical gains are minimal and it complicates the build system. As such, the autotools and CMake tooling was simplified for improved maintainability and this feature was thrown out. clamdtop and clamdscan now require libclamav to function. Removed the nopthreads version of the autotools libclamav_internal_utils static library and added pthread linking to a couple apps that may have issues building on some platforms without it, with the intention of removing needless complexity from the source. Kept the regular version of libclamav_internal_utils.la though it is no longer used anywhere but in libclamav. Added an experimental doxygen build option which attempts to build clamav.h and libfreshclam doxygen html docs. The CMake build tooling also may build the example program(s), which isn't a feature in the Autotools build system. Changed C standard to C90+ due to inline linking issues with socket.h when linking libfreshclam.so on Linux. Generate common.rc for win32. Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef from misc.c. Add CMake files to the automake dist, so users can try the new CMake tooling w/out having to build from a git clone. clamonacc changes: - Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other similar macros. - Added a new clamav-clamonacc.service systemd unit file, based on the work of ChadDevOps & Aaron Brighton. - Added missing clamonacc man page. Updates to clamdscan man page, add missing options. Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use libclamav). Rename Windows mspack.dll to libmspack.dll so all ClamAV-built libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
#include "onas_queue.h"
2019-05-16 13:05:48 -07:00
2019-07-25 12:42:08 -04:00
static void onas_scan_queue_exit(void *arg);
2019-05-16 13:05:48 -07:00
static int onas_consume_event(threadpool thpool);
static cl_error_t onas_new_event_queue_node(struct onas_event_queue_node **node);
static void onas_destroy_event_queue_node(struct onas_event_queue_node *node);
static pthread_mutex_t onas_queue_lock = PTHREAD_MUTEX_INITIALIZER;
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
pthread_cond_t onas_scan_queue_empty_cond = PTHREAD_COND_INITIALIZER;
extern pthread_t scan_queue_pid;
static threadpool g_thpool;
static struct onas_event_queue_node *g_onas_event_queue_head = NULL;
static struct onas_event_queue_node *g_onas_event_queue_tail = NULL;
2019-05-16 13:05:48 -07:00
static struct onas_event_queue g_onas_event_queue;
2019-07-25 12:42:08 -04:00
static cl_error_t onas_new_event_queue_node(struct onas_event_queue_node **node)
{
2019-07-25 12:42:08 -04:00
*node = malloc(sizeof(struct onas_event_queue_node));
if (NULL == *node) {
return CL_EMEM;
}
2019-07-25 12:42:08 -04:00
**node = (struct onas_event_queue_node){
.next = NULL,
.prev = NULL,
2019-07-25 12:42:08 -04:00
.data = NULL};
2019-07-25 12:42:08 -04:00
return CL_SUCCESS;
}
static void *onas_init_event_queue(void)
2019-07-25 12:42:08 -04:00
{
2019-07-25 12:42:08 -04:00
if (CL_EMEM == onas_new_event_queue_node(&g_onas_event_queue_head)) {
return NULL;
}
2019-07-25 12:42:08 -04:00
if (CL_EMEM == onas_new_event_queue_node(&g_onas_event_queue_tail)) {
return NULL;
}
2019-07-25 12:42:08 -04:00
g_onas_event_queue_tail->prev = g_onas_event_queue_head;
g_onas_event_queue_head->next = g_onas_event_queue_tail;
2019-07-25 12:42:08 -04:00
g_onas_event_queue = (struct onas_event_queue){
.head = g_onas_event_queue_head,
.tail = g_onas_event_queue_tail,
2019-07-25 12:42:08 -04:00
.size = 0};
2019-07-25 12:42:08 -04:00
return &g_onas_event_queue;
}
2019-07-25 12:42:08 -04:00
static void onas_destroy_event_queue_node(struct onas_event_queue_node *node)
{
2019-07-25 12:42:08 -04:00
if (NULL == node) {
return;
}
2019-07-25 12:42:08 -04:00
node->next = NULL;
node->prev = NULL;
node->data = NULL;
2019-07-25 12:42:08 -04:00
free(node);
node = NULL;
2019-07-25 12:42:08 -04:00
return;
}
static void onas_destroy_event_queue(void)
2019-07-25 12:42:08 -04:00
{
2019-07-25 12:42:08 -04:00
if (NULL == g_onas_event_queue_head) {
return;
}
2019-07-25 12:42:08 -04:00
struct onas_event_queue_node *curr = g_onas_event_queue_head;
struct onas_event_queue_node *next = curr->next;
2019-07-25 12:42:08 -04:00
do {
onas_destroy_event_queue_node(curr);
curr = next;
if (curr) {
next = curr->next;
}
} while (curr);
2019-07-25 12:42:08 -04:00
return;
}
2019-07-25 12:42:08 -04:00
void *onas_scan_queue_th(void *arg)
{
2023-11-26 15:01:19 -08:00
/* Set thread name for profiling and debugging */
const char thread_name[] = "clamonacc-sq";
#if defined(__linux__)
/* Use prctl instead to prevent using _GNU_SOURCE flag and implicit declaration */
prctl(PR_SET_NAME, thread_name);
#elif defined(__APPLE__) && defined(__MACH__)
pthread_setname_np(thread_name);
#else
logg(LOGG_WARNING, "ClamScanQueue: Setting of the thread name is currently not supported on this system\n");
#endif
2019-07-25 12:42:08 -04:00
/* not a ton of use for context right now, but perhaps in the future we can pass in more options */
struct onas_context *ctx = (struct onas_context *)arg;
sigset_t sigset;
2019-07-25 12:42:08 -04:00
/* ignore all signals except SIGUSR2 */
sigfillset(&sigset);
sigdelset(&sigset, SIGUSR2);
/* The behavior of a process is undefined after it ignores a
* SIGFPE, SIGILL, SIGSEGV, or SIGBUS signal */
2019-07-25 12:42:08 -04:00
sigdelset(&sigset, SIGFPE);
sigdelset(&sigset, SIGILL);
sigdelset(&sigset, SIGSEGV);
sigdelset(&sigset, SIGTERM);
sigdelset(&sigset, SIGINT);
#ifdef SIGBUS
2019-07-25 12:42:08 -04:00
sigdelset(&sigset, SIGBUS);
#endif
pthread_sigmask(SIG_SETMASK, &sigset, NULL);
logg(LOGG_DEBUG, "ClamScanQueue: initializing event queue consumer ... (%d) threads in thread pool\n", ctx->maxthreads);
2019-07-25 12:42:08 -04:00
onas_init_event_queue();
threadpool thpool = thpool_init(ctx->maxthreads);
g_thpool = thpool;
/* loop w/ onas_consume_event until we die */
pthread_cleanup_push(onas_scan_queue_exit, NULL);
logg(LOGG_DEBUG, "ClamScanQueue: waiting to consume events ...\n");
2019-07-25 12:42:08 -04:00
do {
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
onas_consume_event(thpool);
2019-07-25 12:42:08 -04:00
} while (1);
pthread_cleanup_pop(1);
}
static int onas_queue_is_b_empty(void)
2019-07-25 12:42:08 -04:00
{
2019-05-16 13:05:48 -07:00
if (g_onas_event_queue.head->next == g_onas_event_queue.tail) {
return 1;
}
return 0;
}
2019-07-25 12:42:08 -04:00
static int onas_consume_event(threadpool thpool)
{
pthread_mutex_lock(&onas_queue_lock);
while (onas_queue_is_b_empty()) {
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
pthread_cond_wait(&onas_scan_queue_empty_cond, &onas_queue_lock);
}
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
struct onas_event_queue_node *popped_node = g_onas_event_queue_head->next;
g_onas_event_queue_head->next = g_onas_event_queue_head->next->next;
g_onas_event_queue_head->next->prev = g_onas_event_queue_head;
g_onas_event_queue.size--;
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
pthread_mutex_unlock(&onas_queue_lock);
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
thpool_add_work(thpool, (void *)onas_scan_worker, (void *)popped_node->data);
onas_destroy_event_queue_node(popped_node);
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
return 1;
}
2019-07-25 12:42:08 -04:00
cl_error_t onas_queue_event(struct onas_scan_event *event_data)
{
struct onas_event_queue_node *node = NULL;
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
if (CL_EMEM == onas_new_event_queue_node(&node))
2019-07-25 12:42:08 -04:00
return CL_EMEM;
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
pthread_mutex_lock(&onas_queue_lock);
node->next = g_onas_event_queue_tail;
node->prev = g_onas_event_queue_tail->prev;
2019-07-25 12:42:08 -04:00
((struct onas_event_queue_node *)g_onas_event_queue_tail->prev)->next = node;
g_onas_event_queue_tail->prev = node;
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
node->data = event_data;
2019-05-16 13:05:48 -07:00
g_onas_event_queue.size++;
pthread_cond_signal(&onas_scan_queue_empty_cond);
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
pthread_mutex_unlock(&onas_queue_lock);
return CL_SUCCESS;
}
2019-07-25 12:42:08 -04:00
cl_error_t onas_scan_queue_start(struct onas_context **ctx)
{
2019-07-25 12:42:08 -04:00
pthread_attr_t scan_queue_attr;
int32_t thread_started = 1;
2019-07-25 12:42:08 -04:00
if (!ctx || !*ctx) {
logg(LOGG_DEBUG, "ClamScanQueue: unable to start clamonacc. (bad context)\n");
2019-07-25 12:42:08 -04:00
return CL_EARG;
}
2019-07-25 12:42:08 -04:00
if (pthread_attr_init(&scan_queue_attr)) {
return CL_BREAK;
}
pthread_attr_setdetachstate(&scan_queue_attr, PTHREAD_CREATE_JOINABLE);
thread_started = pthread_create(&scan_queue_pid, &scan_queue_attr, onas_scan_queue_th, *ctx);
2019-07-25 12:42:08 -04:00
if (0 != thread_started) {
/* Failed to create thread */
logg(LOGG_DEBUG, "ClamScanQueue: Unable to start event consumer queue thread ... \n");
2019-07-25 12:42:08 -04:00
return CL_ECREAT;
}
2019-07-25 12:42:08 -04:00
return CL_SUCCESS;
}
clamonacc: Reduce warning log verbosity Users have complained about two specific log events that are extremely verbose in non-critical error conditions: - clamonacc reports "ERROR: Can't send to clamd: Bad address" This may occur when small files are created/destroyed before they can be sent to be scanned. The log message probably should only be reported in verbose mode. - clamonacc reports "ClamMisc: $/proc/XXX vanished before UIDs could be excluded; scanning anyway" This may occur when a process that accessed a file exits before clamonacc find out who accessed the file. This is a fairly frequent occurence. It can still be problematic if `clamd` was the process which accessed the file (like a clamd temp file if watching /tmp), generally it's not an issue and we want to silently scan it anyways. Also addressed copypaste issue in onas_send_stream() wherein fd is set to 0 (aka STDIN) if the provided fd == 0 (should've been -1 for invalid FD) and if filename == NULL. In fact clamonacc never scans STDIN so the scan should fail if filename == NULL and the provided FD is invalid (-1). I also found that "Access denied. ERROR" is easily provoked when using --fdpass or --stream using this simple script: for i in {1..5000}; do echo "blah $i" > tmp-$i && rm tmp-$i; done Clamdscan does not allow for scans to fail quietly because the file does not exist, but for clamonacc it's a common thing and we don't want to output an error. To solve this, I changed it so a return length of -1 will still result in an "internal error" message but return len 0 failures will be silently ignored. I've added a static variable to onas_client_scan() that keeps state in case clamd is stopped and started - that way it won't print an error message for every event when offline. Instead it will log an error for the first connection failure, and log again when the connection is re-established for a future scan. Calls to onas_client_scan() are already wrapped with the onas_scan_lock mutex so the static variable should be safe. Finally, there were a couple of error responses from clamd that can occur if the file isn't found which we want to silently ignore, so I've tweaked the code which checks for specific error messages to account for these.
2021-01-03 18:40:48 -08:00
static void onas_scan_queue_exit(void *arg)
2019-07-25 12:42:08 -04:00
{
clamonacc: Reduce warning log verbosity Users have complained about two specific log events that are extremely verbose in non-critical error conditions: - clamonacc reports "ERROR: Can't send to clamd: Bad address" This may occur when small files are created/destroyed before they can be sent to be scanned. The log message probably should only be reported in verbose mode. - clamonacc reports "ClamMisc: $/proc/XXX vanished before UIDs could be excluded; scanning anyway" This may occur when a process that accessed a file exits before clamonacc find out who accessed the file. This is a fairly frequent occurence. It can still be problematic if `clamd` was the process which accessed the file (like a clamd temp file if watching /tmp), generally it's not an issue and we want to silently scan it anyways. Also addressed copypaste issue in onas_send_stream() wherein fd is set to 0 (aka STDIN) if the provided fd == 0 (should've been -1 for invalid FD) and if filename == NULL. In fact clamonacc never scans STDIN so the scan should fail if filename == NULL and the provided FD is invalid (-1). I also found that "Access denied. ERROR" is easily provoked when using --fdpass or --stream using this simple script: for i in {1..5000}; do echo "blah $i" > tmp-$i && rm tmp-$i; done Clamdscan does not allow for scans to fail quietly because the file does not exist, but for clamonacc it's a common thing and we don't want to output an error. To solve this, I changed it so a return length of -1 will still result in an "internal error" message but return len 0 failures will be silently ignored. I've added a static variable to onas_client_scan() that keeps state in case clamd is stopped and started - that way it won't print an error message for every event when offline. Instead it will log an error for the first connection failure, and log again when the connection is re-established for a future scan. Calls to onas_client_scan() are already wrapped with the onas_scan_lock mutex so the static variable should be safe. Finally, there were a couple of error responses from clamd that can occur if the file isn't found which we want to silently ignore, so I've tweaked the code which checks for specific error messages to account for these.
2021-01-03 18:40:48 -08:00
UNUSEDPARAM(arg);
logg(LOGG_DEBUG, "ClamScanQueue: onas_scan_queue_exit()\n");
2019-07-25 12:42:08 -04:00
if (g_thpool) {
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
thpool_wait(g_thpool);
2019-07-25 12:42:08 -04:00
thpool_destroy(g_thpool);
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
g_thpool = NULL;
2019-07-25 12:42:08 -04:00
}
clamonacc: fix possible deadlock in scan queue I faced a problem with tar application hang in case clamonacc is active. Tar is extracting a lot of small files from an archive and stops at some arbitrary point. The problem is not stable to reproduce. I can explain it in the following way: 1. Consumer thread is waiting for condition variable which indicates that there are files in the queue. Once the condition is satisfied, the consumer starts dispatching the files to the worker threads in the thread pool. 2. While the consumer thread processes the files in the queue, producer can put few more items in the queue and fire the condition variable. 3. Consumer thread stucks on waiting for condition variable nevertheless there are items in the queue. However, new items are not coming from tar application because it is waiting on verdict regarding the items in the queue. The solution is to: 1. Use a single mutex to guard the queue and the condition variable 2. Signal condition variable before releasing mutex in producer thread 3. Check if there are events in the queue before waiting for condition Besides, some small things worth to mention: 1. pthread_testcancel() call is not required as pthread_cond_wait() does the check for thread cancelation. 2. Lock seems to be not required in onas_scan_queue_exit() as by this time no one supposed to access the event queue 3. Memory allocation in onas_queue_event() is done without locking the queue 4. Dispatch an item to a worker thread is done without locking the queue (calling thpool_add_work()) Also shutdown ddd thread before event processor thread. This should prevent inserting events to already destroyed queue
2020-11-15 15:28:27 +03:00
onas_destroy_event_queue();
logg(LOGG_INFO, "ClamScanQueue: stopped\n");
}