Commit graph

47 commits

Author SHA1 Message Date
Val S.
23c3cc05f1
Windows: fix number of arguments in function call (#1586)
Some checks failed
clang-format / check-16 (push) Has been cancelled
CMake Build / build-windows (push) Has been cancelled
CMake Build / build-macos (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
CMake Build / build-ubuntu (push) Has been cancelled
clang-format / check (push) Has been cancelled
clang-format / check-1 (push) Has been cancelled
clang-format / check-2 (push) Has been cancelled
clang-format / check-3 (push) Has been cancelled
clang-format / check-4 (push) Has been cancelled
clang-format / check-5 (push) Has been cancelled
clang-format / check-6 (push) Has been cancelled
clang-format / check-7 (push) Has been cancelled
clang-format / check-8 (push) Has been cancelled
clang-format / check-9 (push) Has been cancelled
clang-format / check-10 (push) Has been cancelled
clang-format / check-11 (push) Has been cancelled
clang-format / check-12 (push) Has been cancelled
clang-format / check-13 (push) Has been cancelled
clang-format / check-14 (push) Has been cancelled
clang-format / check-15 (push) Has been cancelled
CodeQL / Analyze-1 (push) Has been cancelled
Fix the number of NULL arguments in `scanmem.c` call to `cl_scandesc_ex()`.
2025-10-04 22:33:01 -04:00
Valerie Snyder
27fe03c751
Fix OpenSSL 1 compatibility issue, plus minor improvements
For OpenSSL 1, `EVP_get_digestbyname()` will fail with "sha2-*" algorithm names.
Must use "sha256", etc.

I made a shim that does the conversion, and I made an improvement to ignore case
when converting alg names to our hash type enumeration.

Other fixes for a few warnings.
2025-08-18 12:27:10 -04:00
Valerie Snyder
0ea66b540a
libclamav: Fix issue reporting trusted verdicts
If the outermost layer is trusted (e.g. using an FP signature), the verdict passed
back by the `cl_scan*_ex()` functions should be CL_VERDICT_TRUSTED.
To ensure this, and other correct verdicts, I moved the logic setting the verdict
to occur when adding indicators, or trusting a given layer. Then at the end of a
scan, it will set the output verdict parameter to the top level verdict.

This commit also:

* Fixes a bug in the `ex_scan_callbacks` program where a crash would happen when
  a hash was retrieved for an inner layer, but isn't for the container.

* Added debug logs whenever a hash is calculated or set, printing the hash type
  and hash string.

* When a layer is trusted, in addition to removing evidence for that layer, it
  will also index the metadata JSON (if that feature is enabled) and will rename
  any "Viruses" to "IgnoredAlerts", and rename "ContainedIndicators" to
  "IgnoredContainedIndicators".

* Fixed an issue where setting the hash algorithm with extra characters, like
  setting to "sha256789" would ignore the extra characters, and report the hash
  type as the same. It will now fail if the string length differs from the known
  hash algorithm.
2025-08-14 22:40:48 -04:00
Valerie Snyder
39fa61869a
Example Program: Add --disable-cache feature
Also minor fixes to sys.rs and clamav.h formatting
2025-08-14 22:40:48 -04:00
Valerie Snyder
ed3e1e55f6
Added additional ex_scan_callbacks test and fixed a couple related bugs
Improvements to the ex_scan_callbacks.c program:
- Print the verdict enum variant names to be more explicit.
- Add the file_props callback (aka metadata JSON) with --gen-json option.
- Add a --debug option.
- Use '-' in option names instead of '_' to be consistent with other programs.
- Add option to disable allmatch, which I named --one-match. :)

Tests: Add ex_scan_callbacks test where --allmatch is disabled.
Verify that CL_VIRUS is returned when a match occurs.
I found a few bugs and inconsistencies from this test and went and fixed
them, and improved the clamav.h function comments as well.
Largely this resulted in cleanup in `cli_magic_scan()` to make sure we
don't accidentally overwrite the return code.
But it also meant making sure that callback functions which are supposed
to trust a file actually clear the evidence/verdict and don't return
CL_VIRUS.
2025-08-14 22:40:47 -04:00
Valerie Snyder
6d9b57eeeb
libclamav: cl_scan*_ex() functions provide verdict separate from errors
It is a shortcoming of existing scan APIs that it is not possible
to return an error without masking a verdict.
We presently work around this limitation by counting up detections at
the end and then overriding the error code with `CL_VIRUS`, if necessary.

The `cl_scanfile_ex()`, `cl_scandesc_ex()`, and `cl_scanmap_ex()` functions
should provide the scan verdict separately from the error code.

This introduces a new enum for recording and reporting a verdict:
`cl_verdict_t` with options:

- `CL_VERDICT_NOTHING_FOUND`
- `CL_VERDICT_TRUSTED`
- `CL_VERDICT_STRONG_INDICATOR`
- `CL_VERDICT_POTENTIALLY_UNWANTED`

Notably, the newer scan APIs may set the verdict to `CL_VERDICT_TRUSTED`
if there is a (hash-based) FP signature for a file, or in the cause where
Authenticode or similar certificate-based verification was performed, or
in the case where an application scan callback returned `CL_VERIFIED`.

CLAM-763
CLAM-865
2025-08-14 22:40:46 -04:00
Valerie Snyder
9d253673f4
Add missing message string when printing CL_BREAK code
Add string message to `cl_strerror()` and in example program for
cl_error_t::CL_BREAK enum variant.

Also fix uninitialized memory use issue in example program.
2025-08-14 22:40:46 -04:00
Valerie Snyder
e223ddb66a
Example program: demonstrate more features and support scripted inputs
Scripted inputs may be used for automated tests.

Added automated tests for the example program to verify correct behavior
using different callback return codes and also using the new scan layer and
fmap API's.

Fixed a bug in ClamAV's evidence module (recording strong, PUA, and
weak indicators for each layer). Rust HashMaps are unordered so the
feature to get the last alert would return a random alert and not
specifically the last one. Switching to IndexMap resolves this, and
allows us to maintain insertion-order for iterating keys even when
removing a key.
2025-08-14 22:40:45 -04:00
Valerie Snyder
4660141186
Auto-format touch-up 2025-08-14 22:39:16 -04:00
Valerie Snyder
13c4788f36
FIPS & FIPS-like limits on hash algs for cryptographic uses
ClamAV will not function when using a FIPS-enabled OpenSSL 3.x.
This is because ClamAV uses MD5 and SHA1 algorithms for a variety of
purposes including matching for malware detection, matching to prevent
false positives on known-clean files, and for verification of MD5-based
RSA digital signatures for determining CVD (signature database archive)
authenticity.

Interestingly, FIPS had been intentionally bypassed when creating hashes
based whole buffers and whole files (by descriptor or `FILE`-pointer):
78d4a9985a
Note: this bypassed FIPS the 1.x way with:
`EVP_MD_CTX_set_flags(ctx, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW);`

It was NOT disabled when using `cl_hash_init()` / `cl_update_hash()` /
`cl_finish_hash()`. That likely worked by coincidence in that the hash
was already calculated most of the time. It certainly would have made
use of those functions if the hash had not been calculated prior:
78d4a9985a/libclamav/matcher.c (L743)

Regardless, bypassing FIPS entirely is not the correct solution.
The FIPS restrictions against using MD5 and SHA1 are valid, particularly
when verifying CVD digital siganatures, but also I think when using a
hash to determine if the file is known-clean (i.e. the "clean cache" and
also MD5-based and SHA1-based FP signatures).

This commit extends the work to bypass FIPS using the newer 3.x method:
`md = EVP_MD_fetch(NULL, alg, "-fips");`

It does this for the legacy `cl_hash*()` functions including
`cl_hash_init()` / `cl_update_hash()` / `cl_finish_hash()`.
It also introduces extended versions that allow the caller to choose if
they want to bypass FIPS:
- `cl_hash_data_ex()`
- `cl_hash_init_ex()`
- `cl_update_hash_ex()`
- `cl_finish_hash_ex()`
- `cl_hash_destroy_ex()`
- `cl_hash_file_fd_ex()`
See the `flags` parameter for each.

Ironically, this commit does NOT use the new functions at this time.
The rational is that ClamAV may need MD5, SHA1, and SHA-256 hashes of
the same files both for determining if the file is malware, and for
determining if the file is clean.

So instead, this commit will do a checks when:

1. Creating a new ClamAV scanning engine. If FIPS-mode enabled, it will
   automatically toggle the "FIPS limits" engine option.
   When loading signatures, if the engine "FIPS limits" option is enabled,
   then MD5 and SHA1 FP signatures will be skipped.

2. Before verifying a CVD (e.g. also for loading, unpacking when
   verification enabled).
   If "FIPS limits" or FIPS-mode are enabled, then the legacy MD5-based RSA
   method is disabled.

   Note: This commit also refactors the interface for `cl_cvdverify_ex()`
   and `cl_cvdunpack_ex()` so they take a `flags` parameters, rather than a
   single `bool`. As these functions are new in this version, it does not
   break the ABI.

The cache was already switched to use SHA2-256, so that's not a concern
for checking FIPS-mode / FIPS limits options.

This adds an option for `freshclam.conf` and `clamd.conf`:

   FIPSCryptoHashLimits yes

And an equivalent command-line option for `clamscan` and `sigtool`:

   --fips-limits

You may programmatically enable FIPS-limits for a ClamAV engine like this:
```C
   cl_engine_set_num(engine, CL_ENGINE_FIPS_LIMITS, 1);
```

CLAM-2792
2025-08-14 22:39:15 -04:00
Valerie Snyder
51adfb8b61
ClamScan & libclamav: improve precision of bytes-scanned, bytes-read
The ClamScan scan summary prints bytes scanned and bytes read in
multiples of 4096 (aka `CL_COUNT_PRECISION`), as is provided by the
`cl_scanfile()`, `cl_scandesc()`, `cl_scanfile_callback()`, and
`cl_scandesc_callback()` functions.

I believe this imprecision was the result of using an `unsigned long int`
which may be 64bit or 32bit, depending on platform. I believe the
intention was to be able to support scanning more than 4 GiB of data.

Since the new `cl_scan*_ex()` functions use a `uint64_t`, which
guarantees a 64bit integer and supports ~16,777,216 terabytes, I find no
reason not to report an accurate count.

For the legacy scan functions (above) I've kept the `CL_COUNT_PRECISION`
behavior to maintain backwards compatibility.

I have also improved the bytes scanned/read output to report GiB, MiB,
KiB, or B as appropriate. Previously, it always report "MB".

CLAM-1433
2025-08-14 22:39:15 -04:00
Valerie Snyder
f05770fb51
libclamav: scan-layer callback API functions
Add the following scan callbacks:
```c
  cl_engine_set_scan_callback(engine, &pre_hash_callback, CL_SCAN_CALLBACK_PRE_HASH);
  cl_engine_set_scan_callback(engine, &pre_scan_callback, CL_SCAN_CALLBACK_PRE_SCAN);
  cl_engine_set_scan_callback(engine, &post_scan_callback, CL_SCAN_CALLBACK_POST_SCAN);
  cl_engine_set_scan_callback(engine, &alert_callback, CL_SCAN_CALLBACK_ALERT);
  cl_engine_set_scan_callback(engine, &file_type_callback, CL_SCAN_CALLBACK_FILE_TYPE);
```

Each callback may alter scan behavior using the following return codes:

* CL_BREAK

  Scan aborted by callback (the rest of the scan is skipped).
  This does not mark the file as clean or infected, it just skips the rest of the scan.

* CL_SUCCESS / CL_CLEAN

  File scan will continue.
  This is different than CL_VERIFIED because it does not affect prior or future alerts.
  Return CL_VERIFIED instead if you want to remove prior alerts for this layer and skip
  the rest of the scan for this layer.

* CL_VIRUS

  This means you don't trust the file. A new alert will be added.
  For CL_SCAN_CALLBACK_ALERT: Means you agree with the alert (no extra alert needed).

* CL_VERIFIED

  Layer explicitly trusted by the callback and previous alerts removed FOR THIS layer.
  You might want to do this if you trust the hash or verified a digital signature.
  The rest of the scan will be skipped FOR THIS layer.
  For contained files, this does NOT mean that the parent or adjacent layers are trusted.

Each callback is given a pointer to the current scan layer from which
they can get previous layers, can get the the layer's fmap, and then
various attributes of the layer and of the fmap such as:
- layer recursion level
- layer object id
- layer file type
- layer attributes (was decerypted, normalized, embedded, or re-typed)
- layer last alert
- fmap name
- fmap hash (md5, sha1, or sha2-256)
- fmap data (pointer and size)
- fmap file descriptor, if any (fd, offset, size)
- fmap filepath, if any (filepath, offset, size)

To make this possible, this commits introduced a handful of new APIs to
query scan-layer details and fmap details:
- `cl_error_t cl_fmap_set_name(cl_fmap_t *map, const char *name);`
- `cl_error_t cl_fmap_get_name(cl_fmap_t *map, const char **name_out);`
- `cl_error_t cl_fmap_set_path(cl_fmap_t *map, const char *path);`
- `cl_error_t cl_fmap_get_path(cl_fmap_t *map, const char **path_out, size_t *offset_out, size_t *len_out);`
- `cl_error_t cl_fmap_get_fd(const cl_fmap_t *map, int *fd_out, size_t *offset_out, size_t *len_out);`
- `cl_error_t cl_fmap_get_size(const cl_fmap_t *map, size_t *size_out);`
- `cl_error_t cl_fmap_set_hash(const cl_fmap_t *map, const char *hash_alg, char hash);`
- `cl_error_t cl_fmap_have_hash(const cl_fmap_t *map, const char *hash_alg, bool *have_hash_out);`
- `cl_error_t cl_fmap_will_need_hash_later(const cl_fmap_t *map, const char *hash_alg);`
- `cl_error_t cl_fmap_get_hash(const cl_fmap_t *map, const char *hash_alg, const char **hash_out);`
- `cl_error_t cl_fmap_get_data(const cl_fmap_t *map, size_t offset, size_t len, const uint8_t **data_out, size_t *data_len_out);`
- `cl_error_t cl_scan_layer_get_fmap(cl_scan_layer_t *layer, cl_fmap_t **fmap_out);`
- `cl_error_t cl_scan_layer_get_parent_layer(cl_scan_layer_t *layer, cl_scan_layer_t **parent_layer_out);`
- `cl_error_t cl_scan_layer_get_type(cl_scan_layer_t *layer, const char **type_out);`
- `cl_error_t cl_scan_layer_get_recursion_level(cl_scan_layer_t *layer, uint32_t *recursion_level_out);`
- `cl_error_t cl_scan_layer_get_object_id(cl_scan_layer_t *layer, uint64_t *object_id_out);`
- `cl_error_t cl_scan_layer_get_last_alert(cl_scan_layer_t *layer, const char **alert_name_out);`
- `cl_error_t cl_scan_layer_get_attributes(cl_scan_layer_t *layer, uint32_t *attributes_out);`

This commit deprecates but does not remove the existing scan callbacks:
- `void cl_engine_set_clcb_pre_cache(struct cl_engine *engine, clcb_pre_cache callback);`
- `void cl_engine_set_clcb_file_inspection(struct cl_engine *engine, clcb_file_inspection callback);`
- `void cl_engine_set_clcb_pre_scan(struct cl_engine *engine, clcb_pre_scan callback);`
- `void cl_engine_set_clcb_post_scan(struct cl_engine *engine, clcb_post_scan callback);`
- `void cl_engine_set_clcb_virus_found(struct cl_engine *engine, clcb_virus_found callback);`
- `void cl_engine_set_clcb_hash(struct cl_engine *engine, clcb_hash callback);`

This commit also adds an interactive test program to demonstrate the callbacks.
See: `examples/ex_scan_callbacks.c`

CLAM-255
CLAM-2485
CLAM-2626
2025-08-14 22:39:14 -04:00
Val Snyder
8d485b9bfd
FIPS-compliant CVD signing and verification
Add X509 certificate chain based signing with PKCS7-PEM external
signatures distributed alongside CVD's in a custom .cvd.sign format.
This new signing and verification mechanism is primarily in support
of FIPS compliance.

Fixes: https://github.com/Cisco-Talos/clamav/issues/564

Add a Rust implementation for parsing, verifying, and unpacking CVD
files.

Now installs a 'certs' directory in the app config directory
(e.g. <prefix>/etc/certs). The install location is configurable.
The CMake option to configure the CVD certs directory is:
  `-D CVD_CERTS_DIRECTORY=PATH`

New options to set an alternative CVD certs directory:
- Commandline for freshclam, clamd, clamscan, and sigtool is:
  `--cvdcertsdir PATH`
- Env variable for freshclam, clamd, clamscan, and sigtool is:
  `CVD_CERTS_DIR`
- Config option for freshclam and clamd is:
  `CVDCertsDirectory PATH`

Sigtool:
- Add sign/verify commands.
- Also verify CDIFF external digital signatures when applying CDIFFs.
- Place commonly used commands at the top of --help string.
- Fix up manpage.

Freshclam:
- Will try to download .sign files to verify CVDs and CDIFFs.
- Fix an issue where making a CLD would only include the CFG file for
daily and not if patching any other database.

libclamav.so:
- Bump version to 13:0:1 (aka 12.1.0).
- Also remove libclamav.map versioning.
  Resolves: https://github.com/Cisco-Talos/clamav/issues/1304
- Add two new API's to the public clamav.h header:
  ```c
  extern cl_error_t cl_cvdverify_ex(const char *file,
                                    const char *certs_directory);

  extern cl_error_t cl_cvdunpack_ex(const char *file,
                                    const char *dir,
                                    bool dont_verify,
                                    const char *certs_directory);
  ```
  The original `cl_cvdverify` and `cl_cvdunpack` are deprecated.
- Add `cl_engine_field` enum option `CL_ENGINE_CVDCERTSDIR`.
  You may set this option with `cl_engine_set_str` and get it
  with `cl_engine_get_str`, to override the compiled in default
  CVD certs directory.

libfreshclam.so: Bump version to 4:0:0 (aka 4.0.0).

Add sigtool sign/verify tests and test certs.

Make it so downloadFile doesn't throw a warning if the server
doesn't have the .sign file.

Replace use of md5-based FP signatures in the unit tests with
sha256-based FP signatures because the md5 implementation used
by Python may be disabled in FIPS mode.
Fixes: https://github.com/Cisco-Talos/clamav/issues/1411

CMake: Add logic to enable the Rust openssl-sys / openssl-rs crates
to build against the same OpenSSL library as is used for the C build.
The Rust unit test application must also link directly with libcrypto
and libssl.

Fix some log messages with missing new lines.

Fix missing environment variable notes in --help messages and manpages.

Deconflict CONFDIR/DATADIR/CERTSDIR variable names that are defined in
clamav-config.h.in for libclamav from variable that had the same name
for use in clamav applications that use the optparser.

The 'clamav-test' certs for the unit tests will live for 10 years.
The 'clamav-beta.crt' public cert will only live for 120 days and will
be replaced before the stable release with a production 'clamav.crt'.
2025-03-26 19:33:25 -04:00
Val Snyder
7ff29b8c37
Bump copyright dates for 2025 2025-02-14 10:24:30 -05:00
Micah Snyder
9cb28e51e6 Bump copyright dates for 2024 2024-01-22 11:27:17 -05:00
Micah Snyder
95df41b5bf Windows: json-c 0.17 compatibility with ssize_t type definition
json-c 0.17 defines the ssize_t type using a typedef on Windows.
We have been setting ssize_t for Windows to a different type using
a #define instead of a typedef.

We should have been using a typedef since it is a type.
However, we must also match the exact type they're setting it to or else
the compiler will baulk because the types are different.

Note: in C11, it's fine to use typedef the same type more than once, so
long as you're defining it the same every time.
2023-08-25 08:56:51 -07:00
Micah Snyder
e27a450bf8
ZIP: Always parse file names
Having the filename is useful for certain callbacks, and will likely be
more useful in the future if we can start comparing detected filetypes
with file extensions.

E.g. if filetype is just "binary" or "text" we may be able to do better
by trusting a ".js" extension to determine the type.
Or else if detected file type is "pe" but the extension is ".png" we may
want to say it's suspicious.

Also adjusted the example callback program to disable metadata option.
The CL_SCAN_GENERAL_COLLECT_METADATA is no longer required for the Zip
parser to record filenames for embedded files, and described in the
previous commit.
This program can be used to demonstrate that it is behaving as desired.
2023-08-23 17:41:40 -07:00
Micah Snyder
38386349c5 Fix many warnings 2023-04-13 00:11:34 -07:00
Micah Snyder
6eebecc303 Bump copyright for 2023 2023-02-12 11:20:22 -08:00
Micah Snyder
d91ae78e99 Rename example programs for readability 2022-10-13 08:57:44 -07:00
Micah Snyder
c7c4ea6063 Examples: demo post-scan callback in file inspection example
Also demo the post-scan callback with alert-encrypted enabled.
2022-10-13 08:57:44 -07:00
Micah Snyder
9d6ebd6d50 Adds file inspection callback and example code
libclamav callbacks can be used to access embedded file content at each
layer of extraction during the course of a scan. The existing callbacks
only provide access to the file descriptor and a guess at the file type.

This patch adds a new callback for the purposes of file/archive
inspection that provides additional insight into the embedded file.
This includes:
- ancestors: an array of parent file names
- parent file size: the size of the direct parent layer
- file name: current layer's filename, if any.
- file buffer (pointer)
- file size: size of file buffer
- file type: just a guess at the current file's type
- file descriptor: may be -1 if the layer is in-memory only.
- layer attributes: a flag field. see LAYER_ATTRIBUTE_* defines in clamav.h

Two new example apps are added that are automatically built when
compiling under CMake:
- ex2 demonstrates the prescan callback.
- ex3 demonstrates the new file inspection callback.

The examples are now installed if enabled, so you can test them in the
Docker image, and so that they'll be colocated with the DLLs so you can
test them on Windows. The installed examples should also be able to find
the UnRAR library at run time, without having to set LD_LIBRARY_PATH.

This commit also sets the fmap->name in an fmap-scan using the basname
of the provided filename if the caller provided the filename and the
provided fmap does not have the name set.
2022-10-13 08:57:44 -07:00
Micah Snyder
a4e6868cea Add example program to test cl_cvdunpack & test 2022-10-12 21:46:54 -07:00
Micah Snyder
8bf70207d5 CMake: Fix LLVM linking issues: libclamav_rust, -ltinfo
We must pass the LLVM library dependencies to the libclamav_rust
build.rs script so it links the libclamav_rust unit test executable with
LLVM.

Also:
- We can remove the libtinfo dependency that was hardcoded for the LLVM
  3.6 support, and must remove it for the build to work on Alpine, macOS.
- Also, increased the libcheck default timeout from 60s to 300s after
  experiencing a failure while testing this.
- Also made one of the valgrind suppressions more generic to account for
  inline optimization differences observed in testing this.
2022-03-09 20:35:42 -08:00
mko-x
a21cc6dcd7
Add explicit log level parameter to application logging API
* Added loglevel parameter to logg()

* Fix logg and mprintf internals with new loglevels

* Update all logg calls to set loglevel

* Update all mprintf calls to set loglevel

* Fix hidden logg calls

* Executed clam-format
2022-02-15 15:13:55 -08:00
micasnyd
140c88aa4e Bump copyright for 2022
Includes minor format corrections.
2022-01-09 14:23:25 -07:00
Micah Snyder (micasnyd)
3573ca810d ClamSubmit: Fix __cfduid cookie failure
Cloudflare deprecated the __cfduid cookie which caused ClamSubmit
failures on systems that stopped receiving the cookie.

This commit removes support for the __cfduid cookie.
Also made the session cookie optional, in case that disappears too.

Changed error messages over to use the logg() function like our other apps.

Tidied up some of the logic, and changed "cleanup" label to "done" to
match other code.
2021-06-22 11:08:22 -07:00
Micah Snyder (micasnyd)
b9ca6ea103 Update copyright dates for 2021
Also fixes up clang-format.
2021-03-19 15:12:26 -07:00
Micah Snyder
afbf0b6180 Fix Windows text file EOL conversion issues
On Windows, files open()'ed without the O_BINARY flag will have new-line
LF (aka \n) converted to CRLF (aka \r\n) automatically when read from or
written to. This is undesirable for all scan targets AND temp files
because it affects pattern matching and with hashing.

This commit converts a handful of instances throughout the codebase
where it appears that O_BINARY was mistakenly omitted and could result
in unexpected behavior on Windows.

Git on Windows also converts LF -> CRLF for "text" files, for editing
purposes.
This is problematic for scan files and test files that should match
verbatim.
We can prevent this issue by marking .ref test files as "binary" in the
.gitattributes file and by always opening scan files and temp files as
binary.

In this commit I've also removed the `ChangeLog merge=cl-merge` line
that was once used to reduce ChangeLog merge conflicts by using the
gnulib git-merge-changlog tool. This project now categorizes changes in
the NEWS.md.
For finer detail, git commit history is fully accessible on github.com.
2021-02-25 11:41:28 -08:00
Micah Snyder
2552cfd0d1 CMake: Add CTest support to match Autotools checks
An ENABLE_TESTS CMake option is provided so that users can disable
testing if they don't want it. Instructions for how to use this
included in the INSTALL.cmake.md file.

If you run `ctest`, each testcase will write out a log file to the
<build>/unit_tests directory.

As with Autotools' make check, the test files are from test/.split
and unit_tests/.split files, but for CMake these are generated at
build time instead of at test time.

On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled
libraries can be loaded when running tests.

On Windows systems, CTest will identify and collect all library
dependencies and assemble a temporarily install under the
build/unit_tests directory so that the libraries can be loaded when
running tests.

The same feature is used on Windows when using CMake to install to
collect all DLL dependencies so that users don't have to install them
manually afterwards.

Each of the CTest tests are run using a custom wrapper around Python's
unittest framework, which is also responsible for finding and inserting
valgrind into the valgrind tests on Posix systems.

Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by
default, if Valgrind can be found. There's no need to set VG=1.
CTest's memcheck module is NOT supported, because we use Python to
orchestrate our tests.

Added a bunch of Windows compatibility changes to the unit tests.
These were primarily changing / to PATHSEP and making adjustments
to use Win32 C headers and ifdef out the POSIX ones which aren't
available on Windows. Also disabled a bunch of tests on Win32
that don't work on Windows, notably the mmap ones and FD-passing
(i.e. FILEDES) ones.

Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate
warnings on Windows where json.h is included after inttypes.h because
json-c's inttypes replacement relies on it.
This is a it of a hack and may be removed if json-c fixes their
inttypes header stuff in the future.

Add preprocessor definitions on Windows to disable MSVC warnings about
CRT secure and nonstandard functions. While there may be a better
solution, this is needed to be able to see other more serious warnings.

Add missing file comment block and copyright statement for clamsubmit.c.
Also change json-c/json.h include filename to json.h in clamsubmit.c.
The directory name is not required.

Changed the hash table data integer type from long, which is poorly
defined, to size_t -- which is capable of storing a pointer. Fixed a
bunch of casts regarding this variable to eliminate warnings.

Fixed two bugs causing utf8 encoding unit tests to fail on Windows:
- The in_size variable should be the number of bytes, not the character
  count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8
  transcoding test to only transcode half the bytes.
- It turns out that the MultiByteToWideChar() API can't transcode
  UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer
  and flip the bytes on each uint16_t. This but was causing the UTF16-BE
  to UTF8 tests to fail.

I also split up the utf8 transcoding tests into separate tests so I
could see all of the failures instead of just the first one.

Added a flags parameter to the unit test function to open testfiles
because it turns out that on Windows if a file contains the \r\n it will
replace it with just \n if you opened the file as a text file instead of
as binary. However, if we open the CBC files as binary, then a bunch of
bytecode tests fail. So I've changed the tests to open the CBC files in
the bytecode tests as text files and open all other files as binary.

Ported the feature tests from shell scripts to Python using a modified
version of our QA test-framework, which is largely compatible and will
allow us to migrate some QA tests into this repo. I'd like to add GitHub
Actions pipelines in the future so that all public PR's get some testing
before anyone has to manually review them.

The clamd --log option was missing from the help string, though it
definitely works. I've added it in this commit.
It appears that clamd.c was never clang-format'd, so this commit also
reformats clamd.c.

Some of the check_clamd tests expected the path returned by clamd to
match character for character with original path sent to clamd. However,
as we now evaluate real paths before a scan, the path returned by clamd
isn't going to match the relative (and possibly symlink-ridden) path
passed to clamdscan. I fixed this test by changing the test to search
for the basename: <signature> FOUND within the response instead of
matching the exact path.

Autotools: Link check_clamd with libclamav so we can use our utility
functions in check_clamd.c.
2021-02-25 11:41:26 -08:00
Micah Snyder (micasnyd)
9e20cdf6ea Add CMake build tooling
This patch adds experimental-quality CMake build tooling.

The libmspack build required a modification to use "" instead of <> for
header #includes. This will hopefully be included in the libmspack
upstream project when adding CMake build tooling to libmspack.

Removed use of libltdl when using CMake.

Flex & Bison are now required to build.

If -DMAINTAINER_MODE, then GPERF is also required, though it currently
doesn't actually do anything.  TODO!

I found that the autotools build system was generating the lexer output
but not actually compiling it, instead using previously generated (and
manually renamed) lexer c source. As a consequence, changes to the .l
and .y files weren't making it into the build. To resolve this, I
removed generated flex/bison files and fixed the tooling to use the
freshly generated files. Flex and bison are now required build tools.
On Windows, this adds a dependency on the winflexbison package,
which can be obtained using Chocolatey or may be manually installed.

CMake tooling only has partial support for building with external LLVM
library, and no support for the internal LLVM (to be removed in the
future). I.e. The CMake build currently only supports the bytecode
interpreter.

Many files used include paths relative to the top source directory or
relative to the current project, rather than relative to each build
target. Modern CMake support requires including internal dependency
headers the same way you would external dependency headers (albeit
with "" instead of <>). This meant correcting all header includes to
be relative to the build targets and not relative to the workspace.

For example, ...

```c
include "../libclamav/clamav.h"
include "clamd/clamd_others.h"
```

... becomes:

```c
// libclamav
include "clamav.h"

// clamd
include "clamd_others.h"
```

Fixes header name conflicts by renaming a few of the files.

Converted the "shared" code into a static library, which depends on
libclamav. The ironically named "shared" static library provides
features common to the ClamAV apps which are not required in
libclamav itself and are not intended for use by downstream projects.
This change was required for correct modern CMake practices but was
also required to use the automake "subdir-objects" option.
This eliminates warnings when running autoreconf which, in the next
version of autoconf & automake are likely to break the build.

libclamav used to build in multiple stages where an earlier stage is
a static library containing utils required by the "shared" code.
Linking clamdscan and clamdtop with this libclamav utils static lib
allowed these two apps to function without libclamav. While this is
nice in theory, the practical gains are minimal and it complicates
the build system. As such, the autotools and CMake tooling was
simplified for improved maintainability and this feature was thrown
out. clamdtop and clamdscan now require libclamav to function.

Removed the nopthreads version of the autotools
libclamav_internal_utils static library and added pthread linking to
a couple apps that may have issues building on some platforms without
it, with the intention of removing needless complexity from the
source. Kept the regular version of libclamav_internal_utils.la
though it is no longer used anywhere but in libclamav.

Added an experimental doxygen build option which attempts to build
clamav.h and libfreshclam doxygen html docs.

The CMake build tooling also may build the example program(s), which
isn't a feature in the Autotools build system.

Changed C standard to C90+ due to inline linking issues with socket.h
when linking libfreshclam.so on Linux.

Generate common.rc for win32.

Fix tabs/spaces in shared Makefile.am, and remove vestigial ifndef
from misc.c.

Add CMake files to the automake dist, so users can try the new
CMake tooling w/out having to build from a git clone.

clamonacc changes:
- Renamed FANOTIFY macro to HAVE_SYS_FANOTIFY_H to better match other
  similar macros.
- Added a new clamav-clamonacc.service systemd unit file, based on
  the work of ChadDevOps & Aaron Brighton.
- Added missing clamonacc man page.

Updates to clamdscan man page, add missing options.

Remove vestigial CL_NOLIBCLAMAV definitions (all apps now use
libclamav).

Rename Windows mspack.dll to libmspack.dll so all ClamAV-built
libraries have the lib-prefix with Visual Studio as with CMake.
2020-08-13 00:25:34 -07:00
Micah Snyder
206dbaefe8 Update copyright dates for 2020 2020-01-03 15:44:07 -05:00
Micah Snyder
20a3dc4273 Adds new clamav-version.h to clamav.h so it doesn't have to be included separately, and adds example usage to the ex1.c example program. 2019-10-02 16:08:30 -04:00
Micah Snyder
52cddcbcfd Updating and cleaning up copyright notices. 2019-10-02 16:08:18 -04:00
Micah Snyder
72fd33c8b2 clang-format'd using new .clang-format rules. 2019-10-02 16:08:16 -04:00
Micah Snyder
38fe8b69a0 Added .clang-format style rules, clam-format script to automate formatting of ClamAV code, and preparing select files so that clang-format does not alter carefully formatted sections. 2019-10-02 16:08:16 -04:00
Micah Snyder (micasnyd)
7e7663abf6 libclamav / clamav.h documentation updated both to clean up existing documentation and to add new documentation. 2018-12-02 23:07:08 -05:00
Micah Snyder
d7979d4ff7 Restructured scan options flags from a single bitflag field to a structure containing multiple bitflag fields. This also required adding a new function to the bytecode API to get scan options a la carte, and modifying the existing function to hand back scan options in the old/deprecated uint32_t bitflag format. Re-generated bytecode iface header files.
Updated libclamav documentation detailing new scan options structure.
Renamed references to 'algorithmic' detection to 'heuristic' detection. Renaming references to 'properties' to 'collect metadata'.
Renamed references to 'scan all' to 'scan all match'.
Renamed a couple of 'Hueristic.*' signature names as 'Heuristics.*' signatures (plural) to match majority of other heuristics.
2018-12-02 23:06:59 -05:00
Mickey Sola
46a35abe56 mass update of copyright headers 2015-09-17 13:41:26 -04:00
Kevin Lin
0945e3c5f1 updated example fileprop analysis bytecodes
moved old example bytecodes to examples/fileprop_analysis/old/
2015-03-04 14:04:45 -05:00
Shawn Webb
641fa9f29b Remove cl_initialize_crypto() call from the example C file 2014-07-04 12:17:33 -04:00
Kevin Lin
b2b7855e00 examples: added fileprop_analysis bytecode source and cud 2014-07-03 13:18:59 -04:00
Shawn Webb
9363412bbb bb11041 - Add cl_initialize_crypto() to the example file 2014-06-20 16:09:28 -04:00
Tomasz Kojm
af90099536 add cl_init(); minor cleanups
git-svn: trunk@4845
2009-02-23 12:20:58 +00:00
Tomasz Kojm
1005682a83 examples/ex1.c: use new API
git-svn: trunk@4842
2009-02-20 14:36:53 +00:00
Tomasz Kojm
50b8f5d66b various updates
git-svn: trunk@3721
2008-03-18 15:40:41 +00:00
Sven Strickroth
a99111f050 remove old CVS-stuff and make the repository look more like SVN
git-svn: trunk@2755
2007-02-17 19:02:20 +00:00