Have indexer apply updates before batch is completed
to ensure that we have an up-to-date index (or we will
if updating fails and we retry).
Rename CustomCacheHandler's methods to avoid confusion.
Close#8902
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Previously MailOfflineCleaner would clean Trash and Spam folders
everytime. This was done under the assumption that Trash and Spam were
excluded from search, which wasn't the case, meaning that they had to be
re-indexed and re-retrieved on every login.
Since we decided to keep Trash and Spam included in search, Trash and
Spam are now cleaned same as all other folders.
Close#8837
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Search extends range on login based on the OfflineTimeRangeDate
Extending search range extends OfflineTimeRangeDate
Co-authored-by: ivk <ivk@tutao.de>
Close#8549
- Introduce a separate Indexer for SQLite using FTS5
- Split search backends and use the right one based on client (IndexedDB
for Browser, and OfflineStorage everywhere else)
- Split SearchFacade into two implementations
- Adds a table for storing unindexed metadata for mails
- Escape special character for SQLite search
To escape special characters from fts5 syntax. However, simply
surrounding each token in quotes is sufficient to do this.
See section 3.1 "FTS5 Strings" here: https://www.sqlite.org/fts5.html
which states that a string may be specified by surrounding it in
quotes, and that special string requirements only exist for strings
that are not in quotes.
- Add EncryptedDbWrapper
- Simplify out of sync logic in IndexedDbIndexer
- Fix deadlock when initializing IndexedDbIndexer
- Cleanup indexedDb index when migrating to offline storage index
- Pass contactSuggestionFacade to IndexedDbSearchFacade
The only suggestion facade used by IndexedDbSearchFacade was the
contact suggestion facade. So we made it clearer.
- Remove IndexerCore stats
- Split custom cache handlers into separate files
We were already doing this with user, so we should do this with the
other entity types.
- Rewrite IndexedDb tests
- Add OfflineStorage indexer tests
- Add custom cache handlers tests to OfflineStorageTest
- Add tests for custom cache handlers with ephemeral storage
- Use dbStub instead of dbMock in IndexedDbIndexerTest
- Replace spy with testdouble in IndexedDbIndexerTest
Close#8550
Co-authored-by: ivk <ivk@tutao.de>
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Co-authored-by: wrd <wrd@tutao.de>
Co-authored-by: bir <bir@tutao.de>
Co-authored-by: hrb-hub <hrb-hub@users.noreply.github.com>
Passing instances explicitly avoids the situations where some of them
might not be initialized.
We also simplified the entity handling by converting entity updates to
data with resolved types early so that the listening code doesn't have
to deal with it.
We did fix some of the bad test practices, e.g. setting/restoring env
incorrectly. This matters now because accessors for type model
initializers check env.mode.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
As we prevent inconsistencies between the data which might be deleted
by the offline cleaner and the range database, we can safely remove
locks.
Co-authored-by: map <mpfau@users.noreply.github.com>
Switches OfflineStorageTest to use in-memory DB, it is faster and less
likely to break (e.g. if DB is corrupted or already exists).
Gets rid of some redundant init() calls that were breaking the db state.
Fixes some assumptions about mails not being deleted.
Refactor our instance deserialization/serialization pipeline, both on
TypeScript and on Rust [sdk] to use typeId and attributeIds instead of
typeNames and attributeNames. We furthermore ignore cardinalities
on associations until the instance layer and always
store associations as arrays. This commit introduces **eventual
consistency** on the client, i.e. we are from now on always storing data
in the newest schema format (activeApplicationVersionsForWritingSum)
which ensures that all data is already available on the client after
updating the client to a newer version. This removes the need for
offline migrations on the client and also removes backward migrations
on the server. Furthermore, the server model types are now available
on the client, retrievable through the ApplicationTypesFacade. This is
our first step towards FastSync.
Co-authored-by: nig <nig@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
When deleting mails MailOfflineCleaner from the OfflineStorage
based on the cutoffId, we were comparing a base64Url encoded customId
of an MailSetEntry with a base64Ext encoded customId cutoffId when
updating the range for the list in OfflineStorage#updateRangeForList.
This lead to wrong ranges for MailSetEntries in the ranges db for
folders with less than 100 mails newer than cutoffId in it.
This commit fixes the comparison, ensuring we compare base64Url.
Furthermore, in order to fix already broken ranges, we modify set lower
range to the cutoffId, in case there are no entities (id == null)
with a lower range currently set to CUSTOM_MIN_ID.
Co-authored-by: abp <abp@tutao.de>
to avoid excessive entity updates and inconsistent offline storages,
we don't send entity updates for each mail set migrated mail.
instead we detect the mail set migration for each folder and drop
its whole mail list from the offline cache.
we could fix up the database when we receive the changed folder,
but that would involve re-doing the migration locally and will
lead to very long entity event handling that might get interrupted,
breaking the offline database anyway.
In order to allow importing of mails we replace legacy MailFolders
(non-static mail listIds) with new MailSets (static mail listIds).
From now on, mails have static mail listIds and static mail elementIds.
To move mails between new MailSets we introduce MailSetEntries
("entries" property on a MailSet), which are index entries sorted by
the received date of the referenced mails (customId). This commit adds
support for new MailSets, while still supporting legacy MailFolders
(mail lists) to support migrating gradually.
* TutanotaModelV74 adds:
* MailSet support
* and defaultAlarmList on GroupSettings
* SystemModelV107 adds model changes for counter (unread mails) updates
* Adapt mail list to show MailSet and legacy mails
The list model is now largely unaware about listIds since it can
display mails from multiple MailBags. MailBags are static mailLists
from which a mail is only removed from when the mail is permanently
deleted.
* Adapt offline storage for mail sets
Offline storage gained the ability to provide cached entities
from a list of ids.
Prior to starting implementing static MailIds and MailSets, we
want to clean up the current TutanotaModel. Therefore, this commit
removes a lot of legacy metamodel definitions that are not used any
longer, including removing the LegacyMailWrapper (legacy mail bodies).
Additionally, this commit inter alia includes:
* removing types no longer needed after migrating to MailDetails, e.g.
the "body", "toRecipients", "ccRecipients", "bccRecipients",
"replyTos", "sentDate" and "headers" references / values from MAIL_TYPE
* removing "mails" reference form MAIL_BOX_TYPE
* removing "subFolders" reference from MAIL_FOLDER
* removing the legacy types MAIL_BODY_TYPE and MAIL_HEADERS
* removing Value.OLD_OWNER_GROUP_NAME, and Value.OLD_AREA_ID_NAME from
FILE_TYPE and CONTACT_TYPE
Closes#7255
Co-authored-by: sug <sug@tutao.de>
Implemented extended notifications which include the sender.
Re-implemented credential encryption and storage, moved credential
data to the native storage, changed credential encryption on mobile to
always be device lock + implement app lock independently of encryption.
Re-implemented SSE on desktop in a more modular way.
Re-organized iOS app to share the code between the main app code and
app extensions.
Close#6608
Co-authored-by: wec43 <wec@tutao.de>
To improve login performance, we want to clear excluded data
(i.e. trash and spam lists, old data) from the offline storage after the login.
We also want to activate "auto_vacuum = incremental" for the offline storage database
in order to allow for faster vacuum execution and run "incremental_vacuum" before
the database is closed.
Fixes#4723
Add ownerGroup column to offline db.
We detect the relationship change by the update of the user. We track
current user in cache storage because it already was receiving it in
case of offline storage and storage already had appropriate lifecycle
for it.
We empty the tables instead of migrating them because we want to remove
them anyway due to corruption (see #4664)
fix#2874
fix#4344
When the database file was corrupted or got out of sync with the key
then it wasn't handled in any way, we would just fall back to the
ephemeral cache.
Now we delete the database and create it again, there's nothing else
we can do.
fix#4382
When adding another account that would override existing stored
credential we would generate a new database key but we would try to
use it with the old database.
Now instead we are forcing the creation of new database and delete
the old one directly from the worker.
When there is a model change on the server that would conflict with
the entities stored in the offline database, we need to do a migration
We generate some model diffs with the code generator and use them
for a compile-time check.
Co-authored-by: ivk <ivk@tutao.de>
2022-05-25 16:30:34 +02:00
Renamed from test/tests/api/worker/rest/OfflineStorageTest.ts (Browse further)
