Passing instances explicitly avoids the situations where some of them
might not be initialized.
We also simplified the entity handling by converting entity updates to
data with resolved types early so that the listening code doesn't have
to deal with it.
We did fix some of the bad test practices, e.g. setting/restoring env
incorrectly. This matters now because accessors for type model
initializers check env.mode.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
As we prevent inconsistencies between the data which might be deleted
by the offline cleaner and the range database, we can safely remove
locks.
Co-authored-by: map <mpfau@users.noreply.github.com>
We introduce ClientTypeModel and ServerTypeModel types and adjust the
usages to make sure server-to-client and client-to-server pipeline uses
the correct types.
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
We change the usages of resolveServerTypeReference to its client
counterpart in all cache functions except for put, as using the server
type reference only makes sense when putting parsed entities in the
cache storage.
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Refactor our instance deserialization/serialization pipeline, both on
TypeScript and on Rust [sdk] to use typeId and attributeIds instead of
typeNames and attributeNames. We furthermore ignore cardinalities
on associations until the instance layer and always
store associations as arrays. This commit introduces **eventual
consistency** on the client, i.e. we are from now on always storing data
in the newest schema format (activeApplicationVersionsForWritingSum)
which ensures that all data is already available on the client after
updating the client to a newer version. This removes the need for
offline migrations on the client and also removes backward migrations
on the server. Furthermore, the server model types are now available
on the client, retrievable through the ApplicationTypesFacade. This is
our first step towards FastSync.
Co-authored-by: nig <nig@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
When deleting mails MailOfflineCleaner from the OfflineStorage
based on the cutoffId, we were comparing a base64Url encoded customId
of an MailSetEntry with a base64Ext encoded customId cutoffId when
updating the range for the list in OfflineStorage#updateRangeForList.
This lead to wrong ranges for MailSetEntries in the ranges db for
folders with less than 100 mails newer than cutoffId in it.
This commit fixes the comparison, ensuring we compare base64Url.
Furthermore, in order to fix already broken ranges, we modify set lower
range to the cutoffId, in case there are no entities (id == null)
with a lower range currently set to CUSTOM_MIN_ID.
Co-authored-by: abp <abp@tutao.de>
Element and list element instances with customIds must be stored with
their elementIds encoded as base64Ext in the offline database, to
ensure that these elementIds are sortable when running SQL queries.
When received from our server customId are encoded with base64Url,
which is not sortable!
When introducing MailSetEntries, which are using
custom elementIds (customId type), we did refactor the OfflineStorage
in such a way that all publicly exposed methods guarantee the correct
encoding before reading / writing from / to the database. With a later
commit f99bd69ff0 some methods in the
OfflineStorage got refactored which led to wrong encodings in certain
scenarios. This commit ensures that custom elementIds are encoded
correctly and are sortable in the OfflineStorage again.
Notification process on mobile does insert Mail instances into offline
db in order to show the mail preview as quickly as possible when the
user clicks on notification. In most cases it happens before login, and
it happens independently of entity update processing.
EntityRestCache does only download new list element instances when
they are within the cached range. In most cases new mails are within the
cached range as mail indexer caches mail bag ranges during initial
indexing. However, if the mail bag size has been reached there will be a
new mail bag with mail list that is not cached by the client (there is
no range stored for this new list).
EventQueue optimizes event updates. If CREATE event is followed by an
UPDATE event they will be folded into a single CREATE event.
In a situation where the email has been updated by another client after
it has been inserted into offline db by notification process but before
the login, and that email belongs to uncached mail bag, CREATE event
(and optimized away UPDATE events) for that mail would be skipped and
the user would see an outdated Mail instance.
We changed the behavior for Mail instances to be always downloaded on
CREATE event when we use Offline cache so that we do not miss those
update operations. This was the previously expected behavior because
of mail indexing and is also a common behavior now, until new mail bag
is created so it does not lead to additional requests in most cases.
Alternatively we could check if the (new) instance is already cached as
individual instance. We decided against that as it adds the delay to
event processing and also does not fit the logic of processing (why
should cache check if new instance is already cache?).
In the future we should try to optimize loading of new instances so
that the sync time does not increase from downloading single emails.
Close#8041
Co-authored-by: ivk <ivk@tutao.de>
to avoid excessive entity updates and inconsistent offline storages,
we don't send entity updates for each mail set migrated mail.
instead we detect the mail set migration for each folder and drop
its whole mail list from the offline cache.
we could fix up the database when we receive the changed folder,
but that would involve re-doing the migration locally and will
lead to very long entity event handling that might get interrupted,
breaking the offline database anyway.
When we try to set correct TypeRef to all aggregates in a type we
traverse the type model. This type model applies to the current version
of the type, not to the one that is in offline db before the migrations
are run.
We separated the typed methods that read the whole table from untyped
ones to avoid the issue above. It has an additional benefit of not
pretending that it returns the entity in its current version.
fix#7524
To sort customIds within the offline database we store customIds
as base64Ext id strings in the offline storage. We want to align
the EphemeralCacheStorage to behave the same, and likewise store
customIds with base64Ext encoding.
To sort customIds within the offline database we store customIds
as base64Ext id strings in the offline storage. Once we retrieve data
from the offline storage we need to convert the customIds to
base64Url respectively, so that other parts can properly compare those
customIds. The only place where customIds are stored / processed in
base64Ext is in the offline database. Everywhere else customIds are
encoded as base64Url id strings.
getIdsInRange is special in the way that it doesn't return parsed
entities, but directly uses the index columns normally only used for
sorting the entities.
we now store custom id entities in the offline storage, which means we need to
make sure storing ranges and comparing ids works for them. in order to achieve
that, we decided to store the normally base64Url-encoded, not lexicographically
sortable ids in the sortable base64Ext format.
the Offline Storage needs to use the "converted" base64Ext ids internally everywhere
for custom id types, but give out ranges and entities in the "raw" base64Url format and
take raw ids as parameters.
to make this easier, we implement the conversion in the public CacheStorage::getRangeForList
implementation and use the private OfflineStorage::getRange method internally.
In order to allow importing of mails we replace legacy MailFolders
(non-static mail listIds) with new MailSets (static mail listIds).
From now on, mails have static mail listIds and static mail elementIds.
To move mails between new MailSets we introduce MailSetEntries
("entries" property on a MailSet), which are index entries sorted by
the received date of the referenced mails (customId). This commit adds
support for new MailSets, while still supporting legacy MailFolders
(mail lists) to support migrating gradually.
* TutanotaModelV74 adds:
* MailSet support
* and defaultAlarmList on GroupSettings
* SystemModelV107 adds model changes for counter (unread mails) updates
* Adapt mail list to show MailSet and legacy mails
The list model is now largely unaware about listIds since it can
display mails from multiple MailBags. MailBags are static mailLists
from which a mail is only removed from when the mail is permanently
deleted.
* Adapt offline storage for mail sets
Offline storage gained the ability to provide cached entities
from a list of ids.
This commit moves some functions that are used in both App and Worker.
Also renames CommonMailUtils to SharedMailUtils.ts because there share
functionalities between both apps and creates a new file CommonMailUtils
to store functions that are common between app and worker.
