We add an event type to EventBusListener to be able to perform the migration in the worker once the synchronization is done.
We send error any to the main thread so the user can reporet them, but we do not stop processing the event queue.
tuta#2079
We should make sure that we never process nor write any entry instance
from an entityUpdate to the offline database in case we encountered
_errors while decrypting the instance. Otherwise, we can end up in
an inconsistent offline database state.
WebSocket messages can contain the ';' character as we include the
instance on the websocket when the client is online now. This will break
the separation logic between the message type and the message. We need
to join them together to make sure the message remains a valid JSON.
We introduce a PrefetchStatus to account for instances not returned
by the server as early as possible. This ensures that we do not throw
in case of 404 not found or 403 not authorized error, when instances
corresponding to create or update entityUpdates have already been
deleted or permissions have been revoked.
Co-authored-by: abp <abp@tutao.de>
When processing the missed entityUpdates in EventQueue in EventBusClient
, we group entityUpdates based on typeRefs and listIds and do
loadMultiple requests instead of loading them one-by-one (prefetching).
Additionally, when the client is online, the server enriches the
WebSocket message with either the instance (in case of a CREATE event),
or with the patches list (in case of an UPDATE event) so that we do not
need to do an additional GET request and can either put the instance
into the cache or update the entry on the cache using the PatchMerger
instead.
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: das <das@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
- Introduce a separate Indexer for SQLite using FTS5
- Split search backends and use the right one based on client (IndexedDB
for Browser, and OfflineStorage everywhere else)
- Split SearchFacade into two implementations
- Adds a table for storing unindexed metadata for mails
- Escape special character for SQLite search
To escape special characters from fts5 syntax. However, simply
surrounding each token in quotes is sufficient to do this.
See section 3.1 "FTS5 Strings" here: https://www.sqlite.org/fts5.html
which states that a string may be specified by surrounding it in
quotes, and that special string requirements only exist for strings
that are not in quotes.
- Add EncryptedDbWrapper
- Simplify out of sync logic in IndexedDbIndexer
- Fix deadlock when initializing IndexedDbIndexer
- Cleanup indexedDb index when migrating to offline storage index
- Pass contactSuggestionFacade to IndexedDbSearchFacade
The only suggestion facade used by IndexedDbSearchFacade was the
contact suggestion facade. So we made it clearer.
- Remove IndexerCore stats
- Split custom cache handlers into separate files
We were already doing this with user, so we should do this with the
other entity types.
- Rewrite IndexedDb tests
- Add OfflineStorage indexer tests
- Add custom cache handlers tests to OfflineStorageTest
- Add tests for custom cache handlers with ephemeral storage
- Use dbStub instead of dbMock in IndexedDbIndexerTest
- Replace spy with testdouble in IndexedDbIndexerTest
Close#8550
Co-authored-by: ivk <ivk@tutao.de>
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Co-authored-by: wrd <wrd@tutao.de>
Co-authored-by: bir <bir@tutao.de>
Co-authored-by: hrb-hub <hrb-hub@users.noreply.github.com>
Passing instances explicitly avoids the situations where some of them
might not be initialized.
We also simplified the entity handling by converting entity updates to
data with resolved types early so that the listening code doesn't have
to deal with it.
We did fix some of the bad test practices, e.g. setting/restoring env
incorrectly. This matters now because accessors for type model
initializers check env.mode.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
We observed clients that managed to get stuck with a missing server
model, which made login from stored credentials impossible and required
getting a new session.
The reason was that only the service executor, rest client and event bus
client cared to update the server model if they were missing.
However, the offline database also stores data in the server's model and
thus requires it to read any data. Since logging in will hit
the cache before requesting the server, a corrupted or missing server
model causes us to not be able to log in.
This commit gives the responsibility of acquiring the models to
ServerModelInfo.resolveServerTypeReference if they are missing or out
of date to avoid situations where we call resolveServerTypeReference
without first checking if the model is initialized.
Additionally, we show a dialog to the user if the models cannot be
initialized instead of an uncaught error.
We considered having the error handler try to repeat model
initialization, but since the original request already failed at that
point, getting back to a normal state would require an extra manual
retry.
fix#9004
Co-authored-by: nig <nig@tutao.de>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: kib <104761667+kibibytium@users.noreply.github.com>
Passing instances explicitly avoids the situations where some of them
might not be initialized.
We also simplified the entity handling by converting entity updates to
data with resolved types early so that the listening code doesn't have
to deal with it.
We did fix some of the bad test practices, e.g. setting/restoring env
incorrectly. This matters now because accessors for type model
initializers check env.mode.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Refactor our instance deserialization/serialization pipeline, both on
TypeScript and on Rust [sdk] to use typeId and attributeIds instead of
typeNames and attributeNames. We furthermore ignore cardinalities
on associations until the instance layer and always
store associations as arrays. This commit introduces **eventual
consistency** on the client, i.e. we are from now on always storing data
in the newest schema format (activeApplicationVersionsForWritingSum)
which ensures that all data is already available on the client after
updating the client to a newer version. This removes the need for
offline migrations on the client and also removes backward migrations
on the server. Furthermore, the server model types are now available
on the client, retrievable through the ApplicationTypesFacade. This is
our first step towards FastSync.
Co-authored-by: nig <nig@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
Calendar subscriptions can collide with entity updates during
application startup. This commit adds a SyncTracker to track the
progress of initial entity updates, and after processing the entire
batch, mark the sync as done, allowing calendars to synchronize with
fewer chance to collisions.
When getting the current group key for a given group, it can be the case
that the group is loaded from the cache before the cache is updated by
the EventBusClient with a new key, so the key that we return is not
actually the current one. Bypassing the cache solves this problem.
It does come with the disadvantage that we may be doing unnecessary
network requests.
tutadb#1883
Previous change to sync progress tracking did not take into account
event optimization where some batches might be skipped, either
immediately or retroactively (e.g. after we receive a delete event).
One change is to not start counting progress until we know how many
batches will be actually processed (so not until they are added to the
queue).
Another change is to not remove empty event batches from the queue to
not invalidate the expected number of batches that we process.
Note that the batches can still be optimized away completely at the
moment when they are added. That's why the caller must check the return
value of `add()`.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
We might receive entityEventUpdates for types that are unknown to the
client and therefore want to exclude these types from being processed.
Therefore, we now remove unknown types that are not in tutanotaTypes
from the eventBatch.
Co-authored-by: nig <nig@tutao.de>
