Passing instances explicitly avoids the situations where some of them
might not be initialized.
We also simplified the entity handling by converting entity updates to
data with resolved types early so that the listening code doesn't have
to deal with it.
We did fix some of the bad test practices, e.g. setting/restoring env
incorrectly. This matters now because accessors for type model
initializers check env.mode.
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Refactor our instance deserialization/serialization pipeline, both on
TypeScript and on Rust [sdk] to use typeId and attributeIds instead of
typeNames and attributeNames. We furthermore ignore cardinalities
on associations until the instance layer and always
store associations as arrays. This commit introduces **eventual
consistency** on the client, i.e. we are from now on always storing data
in the newest schema format (activeApplicationVersionsForWritingSum)
which ensures that all data is already available on the client after
updating the client to a newer version. This removes the need for
offline migrations on the client and also removes backward migrations
on the server. Furthermore, the server model types are now available
on the client, retrievable through the ApplicationTypesFacade. This is
our first step towards FastSync.
Co-authored-by: nig <nig@tutao.de>
Co-authored-by: abp <abp@tutao.de>
Co-authored-by: jomapp <17314077+jomapp@users.noreply.github.com>
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: sug <sug@tutao.de>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
Offline login was broken because RootInstance for TutanotaProperties is
not cached anymore. This happened in 6acc5ff because of confusion about
semantics of isCachedType() which was meant to be used for range
requests only. For non-range requests we should cache all types, except
the explicitly ignored ones.
We fixed the issue by only checking for custom id in range requests.
Close#8396
Co-authored-by: bir <bir@tutao.de>
Change CacheMode.Cache to CacheMode.ReadAndWrite, and CacheMode.Bypass
to CacheMode.WriteOnly.
All three values use the cache, so having one value be called 'Cache'
and another 'Bypass' might lead to the wrong idea of what these fields
actually do in the future, since we now have the new CacheMode.ReadOnly
mode.
Co-authored-by: BijinDev <BijinDev@users.noreply.github.com>
We want to be able to pass custom headers as well as control cache
behavior for ranged requests.
Note that ranged requests do not support not reading from cache at the
moment. This wouldn't be too hard to implement, but the functionality is
not needed for this story.
Closes#8062
Co-authored-by: BijinDev <BijinDev@users.noreply.github.com>
eml and mbox file import in desktop client
pause/resume functionality
chunking for mails and attachments to reduce server requests
testing for imap import source
wisely designed in the alps
Co-authored-by: map <mpfau@users.noreply.github.com>
Co-authored-by: jhm <17314077+jomapp@users.noreply.github.com>
Co-authored-by: Kinan <104761667+kibibytium@users.noreply.github.com>
Co-authored-by: kitsugo <hayashi.jiro@kitsugo.com>
Co-authored-by: nif <nif@tutao.de>
Co-authored-by: sug <sug@tutao.de>
Notification process on mobile does insert Mail instances into offline
db in order to show the mail preview as quickly as possible when the
user clicks on notification. In most cases it happens before login, and
it happens independently of entity update processing.
EntityRestCache does only download new list element instances when
they are within the cached range. In most cases new mails are within the
cached range as mail indexer caches mail bag ranges during initial
indexing. However, if the mail bag size has been reached there will be a
new mail bag with mail list that is not cached by the client (there is
no range stored for this new list).
EventQueue optimizes event updates. If CREATE event is followed by an
UPDATE event they will be folded into a single CREATE event.
In a situation where the email has been updated by another client after
it has been inserted into offline db by notification process but before
the login, and that email belongs to uncached mail bag, CREATE event
(and optimized away UPDATE events) for that mail would be skipped and
the user would see an outdated Mail instance.
We changed the behavior for Mail instances to be always downloaded on
CREATE event when we use Offline cache so that we do not miss those
update operations. This was the previously expected behavior because
of mail indexing and is also a common behavior now, until new mail bag
is created so it does not lead to additional requests in most cases.
Alternatively we could check if the (new) instance is already cached as
individual instance. We decided against that as it adds the delay to
event processing and also does not fit the logic of processing (why
should cache check if new instance is already cache?).
In the future we should try to optimize loading of new instances so
that the sync time does not increase from downloading single emails.
Close#8041
Co-authored-by: ivk <ivk@tutao.de>
After rotating group keys we need to access the former group key for previously encrypted instances.
Former group keys are stored on the GroupKey LET which use a customId. These instances are not
cached by default. But with the recent modification to that cache that allows caching MailSetEntries
GroupKey entites can also be cached. With this commit we enable the caching for this type.
to avoid excessive entity updates and inconsistent offline storages,
we don't send entity updates for each mail set migrated mail.
instead we detect the mail set migration for each folder and drop
its whole mail list from the offline cache.
we could fix up the database when we receive the changed folder,
but that would involve re-doing the migration locally and will
lead to very long entity event handling that might get interrupted,
breaking the offline database anyway.
Show a message if the email to be opened is no longer there.
Keep track of explicitly opened email independent of the list state.
Handle offline errors.
Fix URL and ViewSlider handling
Close#7373
Co-authored-by: jat <jat@tutao.de>
Co-authored-by: paw <paw-hub@users.noreply.github.com>
Open cached emails in the viewer before the list is loaded
Make sure ConversationViewModel displays the primary mail
This is necessary when opening email from notification while being
offline as the mail instance is in the cache from native part but not
the conversation entries.
Co-authored-by: ivk <ivk@tutao.de>
Co-authored-by: jat <jat@tutao.de>
To sort customIds within the offline database we store customIds
as base64Ext id strings in the offline storage. Once we retrieve data
from the offline storage we need to convert the customIds to
base64Url respectively, so that other parts can properly compare those
customIds. The only place where customIds are stored / processed in
base64Ext is in the offline database. Everywhere else customIds are
encoded as base64Url id strings.
getIdsInRange is special in the way that it doesn't return parsed
entities, but directly uses the index columns normally only used for
sorting the entities.
we now store custom id entities in the offline storage, which means we need to
make sure storing ranges and comparing ids works for them. in order to achieve
that, we decided to store the normally base64Url-encoded, not lexicographically
sortable ids in the sortable base64Ext format.
the Offline Storage needs to use the "converted" base64Ext ids internally everywhere
for custom id types, but give out ranges and entities in the "raw" base64Url format and
take raw ids as parameters.
to make this easier, we implement the conversion in the public CacheStorage::getRangeForList
implementation and use the private OfflineStorage::getRange method internally.
In order to allow importing of mails we replace legacy MailFolders
(non-static mail listIds) with new MailSets (static mail listIds).
From now on, mails have static mail listIds and static mail elementIds.
To move mails between new MailSets we introduce MailSetEntries
("entries" property on a MailSet), which are index entries sorted by
the received date of the referenced mails (customId). This commit adds
support for new MailSets, while still supporting legacy MailFolders
(mail lists) to support migrating gradually.
* TutanotaModelV74 adds:
* MailSet support
* and defaultAlarmList on GroupSettings
* SystemModelV107 adds model changes for counter (unread mails) updates
* Adapt mail list to show MailSet and legacy mails
The list model is now largely unaware about listIds since it can
display mails from multiple MailBags. MailBags are static mailLists
from which a mail is only removed from when the mail is permanently
deleted.
* Adapt offline storage for mail sets
Offline storage gained the ability to provide cached entities
from a list of ids.
