We sync the spam training data encrypted through our server to make
sure that all clients for a specific user behave the same when
classifying mails. Additionally, this enables the spam classification
in the webApp. We compress the training data vectors
(see clientSpamTrainingDatum) before uploading to our server using
SparseVectorCompressor.ts. When a user has the ClientSpamClassification
enabled, the spam training data sync will happen for every mail
received.
ClientSpamTrainingDatum are not stored in the CacheStorage.
No entityEvents are emitted for this type.
However, we retrieve creations and updates for ClientSpamTrainingData
through the modifiedClientSpamTrainingDataIndex.
We calculate a threshold per classifier based on the dataset ham to spam
ratio, we also subsample our training data to cap the ham to spam ratio
within a certain limit.
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: das <das@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: nif <nif@tutao.de>
Co-authored-by: map <mpfau@users.noreply.github.com>
Instead of applying inbox rules based on the unread mail state in the
inbox folder, we introduce the new ProcessingState enum on
the mail type. If a mail has been processed by the leader client, which
is checking for matching inbox rules, the ProcessingState is
updated. If there is a matching rule the flag is updated through the
MoveMailService, if there is no matching rule, the flag is updated
using the ClientClassifierResultService. Both requests are
throttled / debounced. After processing inbox rules, spam prediction
is conducted for mails that have not yet been moved by an inbox rule.
The ProcessingState for not matching ham mails is also updated using
the ClientClassifierResultService.
This new inbox rule handing solves the following two problems:
- when clicking on a notification it could still happen,
that sometimes the inbox rules where not applied
- when the inbox folder had a lot of unread mails, the loading time did
massively increase, since inbox rules were re-applied on every load
Co-authored-by: amm <amm@tutao.de>
Co-authored-by: Nick <nif@tutao.de>
Co-authored-by: das <das@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: jhm <17314077+jomapp@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
The error is a result of an incorrect assumption that EntityUpdateData's
instanceListId is null for ElementEntities when processing entity
updates, when in fact instanceListId is an empty string.
Changing EntityUpdateData.instanceListId's type to NonEmptyString | null
and setting it to null for ElementEntity updates early should help
prevent bugs caused by such assumptions.
Close#9420
Co-authored-by: bir <bir@tutao.de>
Co-authored-by: ivk <ivk@tutao.de>
We introduce a PrefetchStatus to account for instances not returned
by the server as early as possible. This ensures that we do not throw
in case of 404 not found or 403 not authorized error, when instances
corresponding to create or update entityUpdates have already been
deleted or permissions have been revoked.
Co-authored-by: abp <abp@tutao.de>
When processing the missed entityUpdates in EventQueue in EventBusClient
, we group entityUpdates based on typeRefs and listIds and do
loadMultiple requests instead of loading them one-by-one (prefetching).
Additionally, when the client is online, the server enriches the
WebSocket message with either the instance (in case of a CREATE event),
or with the patches list (in case of an UPDATE event) so that we do not
need to do an additional GET request and can either put the instance
into the cache or update the entry on the cache using the PatchMerger
instead.
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: das <das@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
We are not relying on the list model to do this, since we might have
older mails get selected. In this case, we were already calling
onSingleSelection, but not in the case where it was the latest mail of
a conversation. We should unconditionally call this.
Closes#9208
Co-authored-by: ivk <ivk@tutao.de>
Make LoadedConversation into a class that manages its own state.
Have the conversation keep track of its mails, not the list model. We
can move a lot of code for mutating LoadedConversations into the
LoadedConversation itself and then call this from the list model.
Also fix how ConversationListModel does updates, since it is memoized
and may not update its arrays when expected.
Closes#9189Closes#9196Closes#9199
Co-authored-by: bir <bir@tutao.de>
If we receive a mailSetEntry create event for a conversation mail that's
already loaded, the entry is inserted into the conversation mails list
without replacing, resulting in duplicates. This causes an inconsistent
state between mailMap and conversation mails when handling mailSetEntry
delete events, which happens because we assume that the conversation
mails list contains unique ids, and therefore we only remove once.
To fix this we simply make sure to avoid duplicates in conversation
mails list when inserting.
Close#9165
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Co-authored-by: ivk <ivk@tutao.de>
When loading of mails fails due to a network error the list model
tries to at least load the cached items. ConversationListModel was
repeatedly trying to load the items from offline because the start
id was not updated when loading items from the cache directly. This
would cause it to load and return the same items in the loop, without
ever throwing an offline error.
We fixed it by making sure the `lastFetchedMailSetEntryId` is set when
loading directly from cache which breaks up the loop.
Fix#9122
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Emails opened through notification would be marked as read before the
inbox rules would apply and this would cause inbox rules to not be
applied.
We changed it by forcibly applying inbox rules to the emails opened
from notifications.
Close#8393
Passing instances explicitly avoids the situations where some of them
might not be initialized.
We also simplified the entity handling by converting entity updates to
data with resolved types early so that the listening code doesn't have
to deal with it.
We did fix some of the bad test practices, e.g. setting/restoring env
incorrectly. This matters now because accessors for type model
initializers check env.mode.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Refactor our instance deserialization/serialization pipeline, both on
TypeScript and on Rust [sdk] to use typeId and attributeIds instead of
typeNames and attributeNames. We furthermore ignore cardinalities
on associations until the instance layer and always
store associations as arrays. This commit introduces **eventual
consistency** on the client, i.e. we are from now on always storing data
in the newest schema format (activeApplicationVersionsForWritingSum)
which ensures that all data is already available on the client after
updating the client to a newer version. This removes the need for
offline migrations on the client and also removes backward migrations
on the server. Furthermore, the server model types are now available
on the client, retrievable through the ApplicationTypesFacade. This is
our first step towards FastSync.
Co-authored-by: nig <nig@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
- This list model groups all emails by conversation in the mail list
- Also adds setting for grouping by conversation (i.e. enables this)
- The setting is overridden for SENT and DRAFT folder types (it uses the
other per-mail list model instead)
- Adds MailSetListModel which provides a common interface for the
MailViewModel so that additional list models can be created; it is
used for MailListModel and the new ConversationListModel, but it can
be extended to even more list models in the future if it is ever
desired.
Note: This does not include actions. Any action you do (delete, archive,
etc.) will be applied to the selected email. The purpose of this commit
is to just implement the list model. See #5051 for more information.
Closes#8223
Co-authored-by: bir <bir@tutao.de>
Co-authored-by: ivk <ivk@tutao.de>
