Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-11-04 07:10:57 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Meta/Lagom/Tools/CodeGenerators/LibUnicode/GenerateUnicodeDateTimeFormat.cpp

2166 lines
84 KiB
C++
Raw Normal View History

LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
/*
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
* Copyright (c) 2021-2022, Tim Flynn <trflynn89@pm.me>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
#include "GeneratorUtil.h"
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
#include <AK/AllOf.h>
#include <AK/CharacterTypes.h>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
#include <AK/Format.h>
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
#include <AK/GenericLexer.h>
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
#include <AK/HashFunctions.h>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
#include <AK/HashMap.h>
#include <AK/JsonObject.h>
#include <AK/JsonParser.h>
#include <AK/JsonValue.h>
#include <AK/LexicalPath.h>
#include <AK/SourceGenerator.h>
#include <AK/String.h>
#include <AK/StringBuilder.h>
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
#include <AK/Traits.h>
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
#include <AK/Utf8View.h>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
#include <LibCore/ArgsParser.h>
#include <LibCore/DirIterator.h>
#include <LibCore/File.h>
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
#include <LibUnicode/DateTimeFormat.h>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
using StringIndexType = u16;
constexpr auto s_string_index_type = "u16"sv;
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
using CalendarPatternIndexType = u16;
constexpr auto s_calendar_pattern_index_type = "u16"sv;
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
using CalendarPatternListIndexType = u8;
constexpr auto s_calendar_pattern_list_index_type = "u8"sv;
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
using CalendarRangePatternIndexType = u16;
constexpr auto s_calendar_range_pattern_index_type = "u16"sv;
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
using CalendarRangePatternListIndexType = u8;
constexpr auto s_calendar_range_pattern_list_index_type = "u8"sv;
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
using CalendarFormatIndexType = u8;
constexpr auto s_calendar_format_index_type = "u8"sv;
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
using SymbolListIndexType = u16;
constexpr auto s_symbol_list_index_type = "u16"sv;
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
using CalendarSymbolsIndexType = u16;
constexpr auto s_calendar_symbols_index_type = "u16"sv;
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
using CalendarSymbolsListIndexType = u8;
constexpr auto s_calendar_symbols_list_index_type = "u8"sv;
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
using CalendarIndexType = u8;
constexpr auto s_calendar_index_type = "u8"sv;
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
using TimeZoneIndexType = u16;
constexpr auto s_time_zone_index_type = "u16"sv;
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
using TimeZoneListIndexType = u8;
constexpr auto s_time_zone_list_index_type = "u8"sv;
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
using TimeZoneFormatIndexType = u8;
constexpr auto s_time_zone_format_index_type = "u8"sv;
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
using DayPeriodIndexType = u8;
constexpr auto s_day_period_index_type = "u8"sv;
using DayPeriodListIndexType = u8;
constexpr auto s_day_period_list_index_type = "u8"sv;
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
using HourCycleListIndexType = u8;
constexpr auto s_hour_cycle_list_index_type = "u8"sv;
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
struct CalendarPattern : public Unicode::CalendarPattern {
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
bool contains_only_date_fields() const
{
return !day_period.has_value() && !hour.has_value() && !minute.has_value() && !second.has_value() && !fractional_second_digits.has_value() && !time_zone_name.has_value();
}
bool contains_only_time_fields() const
{
return !weekday.has_value() && !era.has_value() && !year.has_value() && !month.has_value() && !day.has_value();
}
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
unsigned hash() const
{
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
auto hash = pair_int_hash(pattern_index, pattern12_index);
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
hash = pair_int_hash(hash, skeleton_index);
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
auto hash_field = [&](auto const& field) {
if (field.has_value())
hash = pair_int_hash(hash, static_cast<u8>(*field));
else
hash = pair_int_hash(hash, -1);
};
hash_field(era);
hash_field(year);
hash_field(month);
hash_field(weekday);
hash_field(day);
hash_field(day_period);
hash_field(hour);
hash_field(minute);
hash_field(second);
hash_field(fractional_second_digits);
hash_field(time_zone_name);
return hash;
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
}
bool operator==(CalendarPattern const& other) const
{
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
return (skeleton_index == other.skeleton_index)
&& (pattern_index == other.pattern_index)
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
&& (pattern12_index == other.pattern12_index)
&& (era == other.era)
&& (year == other.year)
&& (month == other.month)
&& (weekday == other.weekday)
&& (day == other.day)
&& (day_period == other.day_period)
&& (hour == other.hour)
&& (minute == other.minute)
&& (second == other.second)
&& (fractional_second_digits == other.fractional_second_digits)
&& (time_zone_name == other.time_zone_name);
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
}
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
StringIndexType skeleton_index { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
StringIndexType pattern_index { 0 };
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
StringIndexType pattern12_index { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
template<>
struct AK::Formatter<CalendarPattern> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, CalendarPattern const& pattern)
{
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
auto field_to_i8 = [](auto const& field) -> i8 {
if (!field.has_value())
return -1;
return static_cast<i8>(*field);
};
return Formatter<FormatString>::format(builder,
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
"{{ {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {} }}",
pattern.skeleton_index,
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
pattern.pattern_index,
pattern.pattern12_index,
field_to_i8(pattern.era),
field_to_i8(pattern.year),
field_to_i8(pattern.month),
field_to_i8(pattern.weekday),
field_to_i8(pattern.day),
field_to_i8(pattern.day_period),
field_to_i8(pattern.hour),
field_to_i8(pattern.minute),
field_to_i8(pattern.second),
field_to_i8(pattern.fractional_second_digits),
field_to_i8(pattern.time_zone_name));
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
}
};
template<>
struct AK::Traits<CalendarPattern> : public GenericTraits<CalendarPattern> {
static unsigned hash(CalendarPattern const& c) { return c.hash(); }
};
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
struct CalendarRangePattern : public CalendarPattern {
unsigned hash() const
{
auto hash = CalendarPattern::hash();
if (field.has_value())
hash = pair_int_hash(hash, static_cast<u8>(*field));
hash = pair_int_hash(hash, start_range);
hash = pair_int_hash(hash, separator);
hash = pair_int_hash(hash, end_range);
return hash;
}
bool operator==(CalendarRangePattern const& other) const
{
if (!CalendarPattern::operator==(other))
return false;
return (field == other.field)
&& (start_range == other.start_range)
&& (separator == other.separator)
&& (end_range == other.end_range);
}
Optional<Unicode::CalendarRangePattern::Field> field {};
StringIndexType start_range { 0 };
StringIndexType separator { 0 };
StringIndexType end_range { 0 };
};
template<>
struct AK::Formatter<CalendarRangePattern> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, CalendarRangePattern const& pattern)
{
auto field_to_i8 = [](auto const& field) -> i8 {
if (!field.has_value())
return -1;
return static_cast<i8>(*field);
};
return Formatter<FormatString>::format(builder,
"{{ {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {} }}",
pattern.skeleton_index,
field_to_i8(pattern.field),
pattern.start_range,
pattern.separator,
pattern.end_range,
field_to_i8(pattern.era),
field_to_i8(pattern.year),
field_to_i8(pattern.month),
field_to_i8(pattern.weekday),
field_to_i8(pattern.day),
field_to_i8(pattern.day_period),
field_to_i8(pattern.hour),
field_to_i8(pattern.minute),
field_to_i8(pattern.second),
field_to_i8(pattern.fractional_second_digits),
field_to_i8(pattern.time_zone_name));
}
};
template<>
struct AK::Traits<CalendarRangePattern> : public GenericTraits<CalendarRangePattern> {
static unsigned hash(CalendarRangePattern const& c) { return c.hash(); }
};
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
struct CalendarFormat {
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
unsigned hash() const
{
auto hash = pair_int_hash(full_format, long_format);
hash = pair_int_hash(hash, medium_format);
hash = pair_int_hash(hash, short_format);
return hash;
}
bool operator==(CalendarFormat const& other) const
{
return (full_format == other.full_format)
&& (long_format == other.long_format)
&& (medium_format == other.medium_format)
&& (short_format == other.short_format);
}
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
CalendarPatternIndexType full_format { 0 };
CalendarPatternIndexType long_format { 0 };
CalendarPatternIndexType medium_format { 0 };
CalendarPatternIndexType short_format { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
template<>
struct AK::Formatter<CalendarFormat> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, CalendarFormat const& pattern)
{
return Formatter<FormatString>::format(builder,
"{{ {}, {}, {}, {} }}",
pattern.full_format,
pattern.long_format,
pattern.medium_format,
pattern.short_format);
}
};
template<>
struct AK::Traits<CalendarFormat> : public GenericTraits<CalendarFormat> {
static unsigned hash(CalendarFormat const& c) { return c.hash(); }
};
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
using SymbolList = Vector<StringIndexType>;
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
struct CalendarSymbols {
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
unsigned hash() const
{
auto hash = pair_int_hash(narrow_symbols, short_symbols);
hash = pair_int_hash(hash, long_symbols);
return hash;
}
bool operator==(CalendarSymbols const& other) const
{
return (narrow_symbols == other.narrow_symbols)
&& (short_symbols == other.short_symbols)
&& (long_symbols == other.long_symbols);
}
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
SymbolListIndexType narrow_symbols { 0 };
SymbolListIndexType short_symbols { 0 };
SymbolListIndexType long_symbols { 0 };
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
template<>
struct AK::Formatter<CalendarSymbols> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, CalendarSymbols const& symbols)
{
return Formatter<FormatString>::format(builder,
"{{ {}, {}, {} }}",
symbols.narrow_symbols,
symbols.short_symbols,
symbols.long_symbols);
}
};
template<>
struct AK::Traits<CalendarSymbols> : public GenericTraits<CalendarSymbols> {
static unsigned hash(CalendarSymbols const& c) { return c.hash(); }
};
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
using CalendarPatternList = Vector<CalendarPatternIndexType>;
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
using CalendarRangePatternList = Vector<CalendarRangePatternIndexType>;
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
using CalendarSymbolsList = Vector<CalendarSymbolsIndexType>;
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
struct Calendar {
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
unsigned hash() const
{
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
auto hash = int_hash(date_formats);
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
hash = pair_int_hash(hash, time_formats);
hash = pair_int_hash(hash, date_time_formats);
hash = pair_int_hash(hash, available_formats);
hash = pair_int_hash(hash, default_range_format);
hash = pair_int_hash(hash, range_formats);
hash = pair_int_hash(hash, range12_formats);
hash = pair_int_hash(hash, symbols);
return hash;
}
bool operator==(Calendar const& other) const
{
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
return (date_formats == other.date_formats)
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
&& (time_formats == other.time_formats)
&& (date_time_formats == other.date_time_formats)
&& (available_formats == other.available_formats)
&& (default_range_format == other.default_range_format)
&& (range_formats == other.range_formats)
&& (range12_formats == other.range12_formats)
&& (symbols == other.symbols);
}
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
CalendarFormatIndexType date_formats { 0 };
CalendarFormatIndexType time_formats { 0 };
CalendarFormatIndexType date_time_formats { 0 };
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
CalendarPatternListIndexType available_formats { 0 };
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
CalendarRangePatternIndexType default_range_format { 0 };
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
CalendarRangePatternListIndexType range_formats { 0 };
CalendarRangePatternListIndexType range12_formats { 0 };
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
CalendarSymbolsListIndexType symbols { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
template<>
struct AK::Formatter<Calendar> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, Calendar const& calendar)
{
return Formatter<FormatString>::format(builder,
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
"{{ {}, {}, {}, {}, {}, {}, {}, {} }}",
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
calendar.date_formats,
calendar.time_formats,
calendar.date_time_formats,
calendar.available_formats,
calendar.default_range_format,
calendar.range_formats,
calendar.range12_formats,
calendar.symbols);
}
};
template<>
struct AK::Traits<Calendar> : public GenericTraits<Calendar> {
static unsigned hash(Calendar const& c) { return c.hash(); }
};
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
struct TimeZone {
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
unsigned hash() const
{
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
return pair_int_hash(long_name, short_name);
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
}
bool operator==(TimeZone const& other) const
{
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
return (long_name == other.long_name) && (short_name == other.short_name);
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
}
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
StringIndexType long_name { 0 };
StringIndexType short_name { 0 };
};
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
template<>
struct AK::Formatter<TimeZone> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, TimeZone const& time_zone)
{
return Formatter<FormatString>::format(builder,
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
"{{ {}, {} }}",
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
time_zone.long_name,
time_zone.short_name);
}
};
template<>
struct AK::Traits<TimeZone> : public GenericTraits<TimeZone> {
static unsigned hash(TimeZone const& t) { return t.hash(); }
};
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
struct TimeZoneFormat {
unsigned hash() const
{
return int_hash(gmt_zero_format);
}
bool operator==(TimeZoneFormat const& other) const
{
return gmt_zero_format == other.gmt_zero_format;
}
StringIndexType gmt_zero_format { 0 };
};
template<>
struct AK::Formatter<TimeZoneFormat> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, TimeZoneFormat const& time_zone_format)
{
return Formatter<FormatString>::format(builder, "{{ {} }}", time_zone_format.gmt_zero_format);
}
};
template<>
struct AK::Traits<TimeZoneFormat> : public GenericTraits<TimeZoneFormat> {
static unsigned hash(TimeZoneFormat const& t) { return t.hash(); }
};
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
struct DayPeriod {
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
unsigned hash() const
{
auto hash = int_hash(static_cast<u8>(day_period));
hash = pair_int_hash(hash, begin);
hash = pair_int_hash(hash, end);
return hash;
}
bool operator==(DayPeriod const& other) const
{
return (day_period == other.day_period)
&& (begin == other.begin)
&& (end == other.end);
}
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
Unicode::DayPeriod day_period {};
u8 begin { 0 };
u8 end { 0 };
};
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
template<>
struct AK::Formatter<DayPeriod> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, DayPeriod const& day_period)
{
return Formatter<FormatString>::format(builder,
"{{ {}, {}, {} }}",
static_cast<u8>(day_period.day_period),
day_period.begin,
day_period.end);
}
};
template<>
struct AK::Traits<DayPeriod> : public GenericTraits<DayPeriod> {
static unsigned hash(DayPeriod const& d) { return d.hash(); }
};
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
using TimeZoneList = Vector<TimeZoneIndexType>;
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
using DayPeriodList = Vector<DayPeriodIndexType>;
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
using HourCycleList = Vector<Unicode::HourCycle>;
template<>
struct AK::Formatter<Unicode::HourCycle> : Formatter<FormatString> {
ErrorOr<void> format(FormatBuilder& builder, Unicode::HourCycle hour_cycle)
{
return Formatter<FormatString>::format(builder, "{}", to_underlying(hour_cycle));
}
};
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
struct Locale {
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
HashMap<String, CalendarIndexType> calendars;
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
TimeZoneListIndexType time_zones { 0 };
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
TimeZoneFormatIndexType time_zone_formats { 0 };
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
DayPeriodListIndexType day_periods { 0 };
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
};
struct UnicodeLocaleData {
UniqueStringStorage<StringIndexType> unique_strings;
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
UniqueStorage<CalendarPattern, CalendarPatternIndexType> unique_patterns;
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
UniqueStorage<CalendarPatternList, CalendarPatternListIndexType> unique_pattern_lists;
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
UniqueStorage<CalendarRangePattern, CalendarRangePatternIndexType> unique_range_patterns;
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
UniqueStorage<CalendarRangePatternList, CalendarRangePatternListIndexType> unique_range_pattern_lists;
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
UniqueStorage<CalendarFormat, CalendarFormatIndexType> unique_formats;
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
UniqueStorage<SymbolList, SymbolListIndexType> unique_symbol_lists;
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
UniqueStorage<CalendarSymbols, CalendarSymbolsIndexType> unique_calendar_symbols;
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
UniqueStorage<CalendarSymbolsList, CalendarSymbolsListIndexType> unique_calendar_symbols_lists;
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
UniqueStorage<Calendar, CalendarIndexType> unique_calendars;
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
UniqueStorage<TimeZone, TimeZoneIndexType> unique_time_zones;
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
UniqueStorage<TimeZoneList, TimeZoneListIndexType> unique_time_zone_lists;
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
UniqueStorage<TimeZoneFormat, TimeZoneFormatIndexType> unique_time_zone_formats;
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
UniqueStorage<DayPeriod, DayPeriodIndexType> unique_day_periods;
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
UniqueStorage<DayPeriodList, DayPeriodListIndexType> unique_day_period_lists;
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
UniqueStorage<HourCycleList, HourCycleListIndexType> unique_hour_cycle_lists;
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
HashMap<String, Locale> locales;
LibUnicode: Hard-code an alias from the Gregorian calendar to Gregory This alias exists because the name "Gregorian" is too long to be used in a locale identifier, i.e. "en-u-ca-gregorian" is invalid. Aliases for calendars are defined here: https://github.com/unicode-org/cldr-json/blob/main/cldr-json/cldr-bcp47/bcp47/calendar.json However, CLDR version 40 neglected to actually include the cldr-bcp47 package in its release, so we don't have access to this data. So for now hard-code this alias so that JavaScript can actually access it. See: https://unicode-org.atlassian.net/browse/CLDR-15158
2021-11-28 09:25:15 -05:00
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
HashMap<String, HourCycleListIndexType> hour_cycles;
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
Vector<String> hour_cycle_regions;
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
HashMap<String, Vector<String>> meta_zones;
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
Vector<String> time_zones { "UTC"sv };
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
Vector<String> calendars;
LibUnicode: Hard-code an alias from the Gregorian calendar to Gregory This alias exists because the name "Gregorian" is too long to be used in a locale identifier, i.e. "en-u-ca-gregorian" is invalid. Aliases for calendars are defined here: https://github.com/unicode-org/cldr-json/blob/main/cldr-json/cldr-bcp47/bcp47/calendar.json However, CLDR version 40 neglected to actually include the cldr-bcp47 package in its release, so we don't have access to this data. So for now hard-code this alias so that JavaScript can actually access it. See: https://unicode-org.atlassian.net/browse/CLDR-15158
2021-11-28 09:25:15 -05:00
Vector<Alias> calendar_aliases {
// FIXME: Aliases should come from BCP47. See: https://unicode-org.atlassian.net/browse/CLDR-15158
{ "gregorian"sv, "gregory"sv },
};
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
};
LibUnicode: Parse and generate secondary day period symbols Generate morning2, afternoon2, evening2, and night2 symbols.
2021-12-10 11:59:58 -05:00
static Optional<Unicode::DayPeriod> day_period_from_string(StringView day_period)
{
if (day_period == "am"sv)
return Unicode::DayPeriod::AM;
if (day_period == "pm"sv)
return Unicode::DayPeriod::PM;
if (day_period == "morning1"sv)
return Unicode::DayPeriod::Morning1;
if (day_period == "morning2"sv)
return Unicode::DayPeriod::Morning2;
if (day_period == "afternoon1"sv)
return Unicode::DayPeriod::Afternoon1;
if (day_period == "afternoon2"sv)
return Unicode::DayPeriod::Afternoon2;
if (day_period == "evening1"sv)
return Unicode::DayPeriod::Evening1;
if (day_period == "evening2"sv)
return Unicode::DayPeriod::Evening2;
if (day_period == "night1"sv)
return Unicode::DayPeriod::Night1;
if (day_period == "night2"sv)
return Unicode::DayPeriod::Night2;
return {};
};
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
static ErrorOr<void> parse_hour_cycles(String core_path, UnicodeLocaleData& locale_data)
{
// https://unicode.org/reports/tr35/tr35-dates.html#Time_Data
LexicalPath time_data_path(move(core_path));
time_data_path = time_data_path.append("supplemental"sv);
time_data_path = time_data_path.append("timeData.json"sv);
auto time_data_file = TRY(Core::File::open(time_data_path.string(), Core::OpenMode::ReadOnly));
auto time_data = TRY(JsonValue::from_string(time_data_file->read_all()));
auto const& supplemental_object = time_data.as_object().get("supplemental"sv);
auto const& time_data_object = supplemental_object.as_object().get("timeData"sv);
auto parse_hour_cycle = [](StringView hour_cycle) -> Optional<Unicode::HourCycle> {
if (hour_cycle == "h"sv)
return Unicode::HourCycle::H12;
if (hour_cycle == "H"sv)
return Unicode::HourCycle::H23;
if (hour_cycle == "K"sv)
return Unicode::HourCycle::H11;
if (hour_cycle == "k"sv)
return Unicode::HourCycle::H24;
return {};
};
time_data_object.as_object().for_each_member([&](auto const& key, JsonValue const& value) {
auto allowed_hour_cycles_string = value.as_object().get("_allowed"sv).as_string();
auto allowed_hour_cycles = allowed_hour_cycles_string.split_view(' ');
Vector<Unicode::HourCycle> hour_cycles;
for (auto allowed_hour_cycle : allowed_hour_cycles) {
if (auto hour_cycle = parse_hour_cycle(allowed_hour_cycle); hour_cycle.has_value())
hour_cycles.append(*hour_cycle);
}
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
auto hour_cycles_index = locale_data.unique_hour_cycle_lists.ensure(move(hour_cycles));
locale_data.hour_cycles.set(key, hour_cycles_index);
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
if (!locale_data.hour_cycle_regions.contains_slow(key))
locale_data.hour_cycle_regions.append(key);
});
return {};
};
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
static ErrorOr<void> parse_meta_zones(String core_path, UnicodeLocaleData& locale_data)
{
// https://unicode.org/reports/tr35/tr35-dates.html#Metazones
LexicalPath meta_zone_path(move(core_path));
meta_zone_path = meta_zone_path.append("supplemental"sv);
meta_zone_path = meta_zone_path.append("metaZones.json"sv);
auto meta_zone_file = TRY(Core::File::open(meta_zone_path.string(), Core::OpenMode::ReadOnly));
auto meta_zone = TRY(JsonValue::from_string(meta_zone_file->read_all()));
auto const& supplemental_object = meta_zone.as_object().get("supplemental"sv);
auto const& meta_zone_object = supplemental_object.as_object().get("metaZones"sv);
auto const& meta_zone_array = meta_zone_object.as_object().get("metazones"sv);
meta_zone_array.as_array().for_each([&](JsonValue const& value) {
auto const& mapping = value.as_object().get("mapZone"sv);
auto const& meta_zone = mapping.as_object().get("_other"sv);
auto const& golden_zone = mapping.as_object().get("_type"sv);
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
auto& golden_zones = locale_data.meta_zones.ensure(meta_zone.as_string());
golden_zones.append(golden_zone.as_string());
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
});
// UTC does not appear in metaZones.json. Define it for convenience so other parsers don't need to check for its existence.
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
locale_data.meta_zones.set("UTC"sv, { "UTC"sv });
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
return {};
};
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
static constexpr auto is_char(char ch)
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
{
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
return [ch](auto c) { return c == ch; };
}
// For patterns that are 12-hour aware, we need to generate two patterns: one with the day period
// (e.g. {ampm}) in the pattern, and one without the day period. We need to take care to remove
// extra spaces around the day period. Some example expected removals:
//
// "{hour}:{minute} {ampm}" becomes "{hour}:{minute}" (remove the space before {ampm})
// "{ampm} {hour}" becomes "{hour}" (remove the space after {ampm})
// "{hour}:{minute} {ampm} {timeZoneName}" becomes "{hour}:{minute} {timeZoneName}" (remove one of the spaces around {ampm})
static String remove_period_from_pattern(String pattern)
{
for (auto remove : AK::Array { "({ampm})"sv, "{ampm}"sv, "({dayPeriod})"sv, "{dayPeriod}"sv }) {
auto index = pattern.find(remove);
if (!index.has_value())
continue;
constexpr u32 space = ' ';
constexpr u32 open = '{';
constexpr u32 close = '}';
Utf8View utf8_pattern { pattern };
Optional<u32> before_removal;
Optional<u32> after_removal;
for (auto it = utf8_pattern.begin(); utf8_pattern.byte_offset_of(it) < *index; ++it)
before_removal = *it;
if (auto it = utf8_pattern.iterator_at_byte_offset(*index + remove.length()); it != utf8_pattern.end())
after_removal = *it;
if ((before_removal == space) && (after_removal != open)) {
pattern = String::formatted("{}{}",
pattern.substring_view(0, *index - 1),
pattern.substring_view(*index + remove.length()));
} else if ((after_removal == space) && (before_removal != close)) {
pattern = String::formatted("{}{}",
pattern.substring_view(0, *index),
pattern.substring_view(*index + remove.length() + 1));
} else {
pattern = String::formatted("{}{}",
pattern.substring_view(0, *index),
pattern.substring_view(*index + remove.length()));
}
}
return pattern;
}
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
static Optional<CalendarPattern> parse_date_time_pattern_raw(String pattern, String skeleton, UnicodeLocaleData& locale_data)
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
{
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
// https://unicode.org/reports/tr35/tr35-dates.html#Date_Field_Symbol_Table
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
using Unicode::CalendarPatternStyle;
CalendarPattern format {};
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
if (!skeleton.is_empty())
format.skeleton_index = locale_data.unique_strings.ensure(move(skeleton));
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
GenericLexer lexer { pattern };
StringBuilder builder;
bool hour12 { false };
while (!lexer.is_eof()) {
// Literal strings enclosed by quotes are to be appended to the pattern as-is without further
// processing (this just avoids conflicts with the patterns below).
if (lexer.next_is(is_quote)) {
builder.append(lexer.consume_quoted_string());
continue;
}
auto starting_char = lexer.peek();
auto segment = lexer.consume_while([&](char ch) { return ch == starting_char; });
// Era
if (all_of(segment, is_char('G'))) {
builder.append("{era}");
if (segment.length() <= 3)
format.era = CalendarPatternStyle::Short;
else if (segment.length() == 4)
format.era = CalendarPatternStyle::Long;
else
format.era = CalendarPatternStyle::Narrow;
}
// Year
else if (all_of(segment, is_any_of("yYuUr"sv))) {
builder.append("{year}");
if (segment.length() == 2)
format.year = CalendarPatternStyle::TwoDigit;
else
format.year = CalendarPatternStyle::Numeric;
}
// Quarter
else if (all_of(segment, is_any_of("qQ"sv))) {
// Intl.DateTimeFormat does not support quarter formatting, so drop these patterns.
return {};
}
// Month
else if (all_of(segment, is_any_of("ML"sv))) {
builder.append("{month}");
if (segment.length() == 1)
format.month = CalendarPatternStyle::Numeric;
else if (segment.length() == 2)
format.month = CalendarPatternStyle::TwoDigit;
else if (segment.length() == 3)
format.month = CalendarPatternStyle::Short;
else if (segment.length() == 4)
format.month = CalendarPatternStyle::Long;
else if (segment.length() == 5)
format.month = CalendarPatternStyle::Narrow;
} else if (all_of(segment, is_char('l'))) {
// Using 'l' for month formatting is deprecated by TR-35, ensure it is not used.
return {};
}
// Week
else if (all_of(segment, is_any_of("wW"sv))) {
// Intl.DateTimeFormat does not support week formatting, so drop these patterns.
return {};
}
// Day
else if (all_of(segment, is_char('d'))) {
builder.append("{day}");
if (segment.length() == 1)
format.day = CalendarPatternStyle::Numeric;
else
format.day = CalendarPatternStyle::TwoDigit;
LibUnicode: Fix typo in format pattern parser See: https://unicode.org/reports/tr35/tr35-dates.html#dfst-day
2021-12-08 14:50:29 -05:00
} else if (all_of(segment, is_any_of("DFg"sv))) {
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
builder.append("{day}");
format.day = CalendarPatternStyle::Numeric;
}
// Weekday
else if (all_of(segment, is_char('E'))) {
builder.append("{weekday}");
if (segment.length() == 4)
format.weekday = CalendarPatternStyle::Long;
else if (segment.length() == 5)
format.weekday = CalendarPatternStyle::Narrow;
else
format.weekday = CalendarPatternStyle::Short;
} else if (all_of(segment, is_any_of("ec"sv))) {
builder.append("{weekday}");
// TR-35 defines "e", "c", and "cc" as as numeric, and "ee" as 2-digit, but those
// pattern styles are not supported by Intl.DateTimeFormat.
if (segment.length() <= 2)
return {};
if (segment.length() == 4)
format.weekday = CalendarPatternStyle::Long;
else if (segment.length() == 5)
format.weekday = CalendarPatternStyle::Narrow;
else
format.weekday = CalendarPatternStyle::Short;
}
// Period
else if (all_of(segment, is_any_of("ab"sv))) {
builder.append("{ampm}");
hour12 = true;
} else if (all_of(segment, is_char('B'))) {
builder.append("{dayPeriod}");
hour12 = true;
if (segment.length() == 4)
format.day_period = CalendarPatternStyle::Long;
else if (segment.length() == 5)
format.day_period = CalendarPatternStyle::Narrow;
else
format.day_period = CalendarPatternStyle::Short;
}
// Hour
else if (all_of(segment, is_any_of("hHKk"sv))) {
builder.append("{hour}");
if ((segment[0] == 'h') || (segment[0] == 'K'))
hour12 = true;
if (segment.length() == 1)
format.hour = CalendarPatternStyle::Numeric;
else
format.hour = CalendarPatternStyle::TwoDigit;
} else if (all_of(segment, is_any_of("jJC"sv))) {
// TR-35 indicates these should not be used.
return {};
}
// Minute
else if (all_of(segment, is_char('m'))) {
builder.append("{minute}");
if (segment.length() == 1)
format.minute = CalendarPatternStyle::Numeric;
else
format.minute = CalendarPatternStyle::TwoDigit;
}
// Second
else if (all_of(segment, is_char('s'))) {
builder.append("{second}");
if (segment.length() == 1)
format.second = CalendarPatternStyle::Numeric;
else
format.second = CalendarPatternStyle::TwoDigit;
} else if (all_of(segment, is_char('S'))) {
builder.append("{fractionalSecondDigits}");
VERIFY(segment.length() <= 3);
format.fractional_second_digits = static_cast<u8>(segment.length());
} else if (all_of(segment, is_char('A'))) {
// Intl.DateTimeFormat does not support millisecond formatting, so drop these patterns.
return {};
}
// Zone
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
else if (all_of(segment, is_any_of("zV"sv))) {
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
builder.append("{timeZoneName}");
if (segment.length() < 4)
format.time_zone_name = CalendarPatternStyle::Short;
else
format.time_zone_name = CalendarPatternStyle::Long;
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
} else if (all_of(segment, is_any_of("ZOXx"sv))) {
builder.append("{timeZoneName}");
if (segment.length() < 4)
format.time_zone_name = CalendarPatternStyle::ShortOffset;
else
format.time_zone_name = CalendarPatternStyle::LongOffset;
} else if (all_of(segment, is_char('v'))) {
builder.append("{timeZoneName}");
if (segment.length() < 4)
format.time_zone_name = CalendarPatternStyle::ShortGeneric;
else
format.time_zone_name = CalendarPatternStyle::LongGeneric;
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
}
// Non-patterns
else {
builder.append(segment);
}
}
pattern = builder.build();
if (hour12) {
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
format.pattern = remove_period_from_pattern(pattern);
format.pattern12 = move(pattern);
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
} else {
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
format.pattern = move(pattern);
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
}
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
return format;
}
static Optional<CalendarPatternIndexType> parse_date_time_pattern(String pattern, String skeleton, UnicodeLocaleData& locale_data)
{
auto format = parse_date_time_pattern_raw(move(pattern), move(skeleton), locale_data);
if (!format.has_value())
return {};
format->pattern_index = locale_data.unique_strings.ensure(move(format->pattern));
if (format->pattern12.has_value())
format->pattern12_index = locale_data.unique_strings.ensure(format->pattern12.release_value());
return locale_data.unique_patterns.ensure(format.release_value());
}
template<typename... Chars>
static constexpr bool char_is_one_of(char ch, Chars&&... chars)
{
return ((ch == chars) || ...);
}
static void parse_interval_patterns(Calendar& calendar, JsonObject const& interval_formats_object, UnicodeLocaleData& locale_data)
{
// https://unicode.org/reports/tr35/tr35-dates.html#intervalFormats
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
CalendarRangePatternList range_formats {};
CalendarRangePatternList range12_formats {};
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
auto name_of_field = [&](char field) {
if (char_is_one_of(field, 'G'))
return Unicode::CalendarRangePattern::Field::Era;
if (char_is_one_of(field, 'y', 'Y', 'u', 'U', 'r'))
return Unicode::CalendarRangePattern::Field::Year;
if (char_is_one_of(field, 'M', 'L'))
return Unicode::CalendarRangePattern::Field::Month;
if (char_is_one_of(field, 'd', 'D', 'F', 'g'))
return Unicode::CalendarRangePattern::Field::Day;
if (char_is_one_of(field, 'a', 'b'))
return Unicode::CalendarRangePattern::Field::AmPm;
if (char_is_one_of(field, 'B'))
return Unicode::CalendarRangePattern::Field::DayPeriod;
if (char_is_one_of(field, 'h', 'H', 'K', 'k'))
return Unicode::CalendarRangePattern::Field::Hour;
if (char_is_one_of(field, 'm'))
return Unicode::CalendarRangePattern::Field::Minute;
if (char_is_one_of(field, 's'))
return Unicode::CalendarRangePattern::Field::Second;
if (char_is_one_of(field, 'S'))
return Unicode::CalendarRangePattern::Field::FractionalSecondDigits;
VERIFY_NOT_REACHED();
};
auto split_default_range_pattern = [&](auto skeleton, auto const& pattern) {
auto start_range_end = pattern.find('}').value() + 1;
auto end_range_begin = pattern.find_last('{').value();
auto start_range = pattern.substring_view(0, start_range_end);
auto separator = pattern.substring_view(start_range_end, end_range_begin - start_range_end);
auto end_range = pattern.substring_view(end_range_begin);
CalendarRangePattern format {};
format.skeleton_index = locale_data.unique_strings.ensure(skeleton);
format.start_range = locale_data.unique_strings.ensure(start_range);
format.separator = locale_data.unique_strings.ensure(separator);
format.end_range = locale_data.unique_strings.ensure(end_range);
return format;
};
auto split_range_pattern = [&](auto skeleton, auto field, auto const& pattern, auto const& parsed_fields) {
HashMap<StringView, size_t> partitions;
StringView last_partition;
auto begin_index = pattern.find('{');
size_t end_index = 0;
while (begin_index.has_value()) {
end_index = pattern.find('}', *begin_index).value();
auto partition = pattern.substring_view(*begin_index, end_index - *begin_index);
if (partitions.contains(partition))
break;
partitions.set(partition, *begin_index);
last_partition = partition;
begin_index = pattern.find('{', end_index + 1);
}
VERIFY(begin_index.has_value() && !last_partition.is_empty());
auto start_range_end = partitions.get(last_partition).value() + last_partition.length() + 1;
auto start_range = pattern.substring_view(0, start_range_end);
auto separator = pattern.substring_view(start_range_end, *begin_index - start_range_end);
auto end_range = pattern.substring_view(*begin_index);
CalendarRangePattern format {};
format.skeleton_index = locale_data.unique_strings.ensure(skeleton);
format.field = field;
format.start_range = locale_data.unique_strings.ensure(start_range);
format.separator = locale_data.unique_strings.ensure(separator);
format.end_range = locale_data.unique_strings.ensure(end_range);
format.for_each_calendar_field_zipped_with(parsed_fields, [](auto& format_field, auto const& parsed_field, auto) {
format_field = parsed_field;
});
return format;
};
interval_formats_object.for_each_member([&](auto const& skeleton, auto const& value) {
if (skeleton == "intervalFormatFallback"sv) {
auto range_format = split_default_range_pattern(skeleton, value.as_string());
calendar.default_range_format = locale_data.unique_range_patterns.ensure(move(range_format));
return;
}
value.as_object().for_each_member([&](auto const& field, auto const& pattern) {
if (field.ends_with("alt-variant"sv))
return;
VERIFY(field.length() == 1);
auto name = name_of_field(field[0]);
auto format = parse_date_time_pattern_raw(pattern.as_string(), skeleton, locale_data).release_value();
auto range_format = split_range_pattern(skeleton, name, format.pattern, format);
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
range_formats.append(locale_data.unique_range_patterns.ensure(move(range_format)));
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (format.pattern12.has_value()) {
auto range12_pattern = split_range_pattern(skeleton, name, *format.pattern12, format);
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
range12_formats.append(locale_data.unique_range_patterns.ensure(move(range12_pattern)));
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
} else {
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
range12_formats.append(range_formats.last());
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
}
});
});
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
calendar.range_formats = locale_data.unique_range_pattern_lists.ensure(move(range_formats));
calendar.range12_formats = locale_data.unique_range_pattern_lists.ensure(move(range12_formats));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
}
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
static void generate_missing_patterns(Calendar& calendar, CalendarPatternList& formats, Vector<CalendarPattern> date_formats, Vector<CalendarPattern> time_formats, UnicodeLocaleData& locale_data)
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
{
// https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields
auto replace_pattern = [&](auto format, auto time_format, auto date_format) {
auto pattern = locale_data.unique_strings.get(format);
auto time_pattern = locale_data.unique_strings.get(time_format);
auto date_pattern = locale_data.unique_strings.get(date_format);
auto new_pattern = pattern.replace("{0}", time_pattern).replace("{1}", date_pattern);
return locale_data.unique_strings.ensure(move(new_pattern));
};
LibJS+LibUnicode: Generate missing patterns with fractionalSecondDigits TR-35's Matching Skeleton algorithm dictates how user requests including fractional second digits should be handled when the CLDR format pattern does not include that field. When the format pattern contains {second}, but does not contain {fractionalSecondDigits}, generate a second pattern which appends "{decimal}{fractionalSecondDigits}" to the {second} field.
2021-12-06 22:50:54 -05:00
auto inject_fractional_second_digits = [&](auto format) {
auto pattern = locale_data.unique_strings.get(format);
auto new_pattern = pattern.replace("{second}"sv, "{second}{decimal}{fractionalSecondDigits}"sv);
return locale_data.unique_strings.ensure(move(new_pattern));
};
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
auto append_if_unique = [&](auto format) {
auto format_index = locale_data.unique_patterns.ensure(move(format));
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
if (!formats.contains_slow(format_index))
formats.append(format_index);
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
};
LibJS+LibUnicode: Generate missing patterns with fractionalSecondDigits TR-35's Matching Skeleton algorithm dictates how user requests including fractional second digits should be handled when the CLDR format pattern does not include that field. When the format pattern contains {second}, but does not contain {fractionalSecondDigits}, generate a second pattern which appends "{decimal}{fractionalSecondDigits}" to the {second} field.
2021-12-06 22:50:54 -05:00
Vector<CalendarPattern> time_formats_with_fractional_second_digits;
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
for (auto const& format : date_formats)
append_if_unique(format);
LibJS+LibUnicode: Generate missing patterns with fractionalSecondDigits TR-35's Matching Skeleton algorithm dictates how user requests including fractional second digits should be handled when the CLDR format pattern does not include that field. When the format pattern contains {second}, but does not contain {fractionalSecondDigits}, generate a second pattern which appends "{decimal}{fractionalSecondDigits}" to the {second} field.
2021-12-06 22:50:54 -05:00
for (auto const& format : time_formats) {
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
append_if_unique(format);
LibJS+LibUnicode: Generate missing patterns with fractionalSecondDigits TR-35's Matching Skeleton algorithm dictates how user requests including fractional second digits should be handled when the CLDR format pattern does not include that field. When the format pattern contains {second}, but does not contain {fractionalSecondDigits}, generate a second pattern which appends "{decimal}{fractionalSecondDigits}" to the {second} field.
2021-12-06 22:50:54 -05:00
if (format.second.has_value() && !format.fractional_second_digits.has_value()) {
auto new_format = format;
new_format.fractional_second_digits = 2;
new_format.pattern_index = inject_fractional_second_digits(new_format.pattern_index);
if (new_format.pattern12_index != 0)
new_format.pattern12_index = inject_fractional_second_digits(new_format.pattern12_index);
time_formats_with_fractional_second_digits.append(new_format);
append_if_unique(move(new_format));
}
}
time_formats.extend(move(time_formats_with_fractional_second_digits));
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
for (auto const& date_format : date_formats) {
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
auto const& date_time_formats = locale_data.unique_formats.get(calendar.date_time_formats);
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
CalendarPatternIndexType date_time_format_index = 0;
if (date_format.month == Unicode::CalendarPatternStyle::Long) {
if (date_format.weekday.has_value())
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
date_time_format_index = date_time_formats.full_format;
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
else
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
date_time_format_index = date_time_formats.long_format;
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
} else if (date_format.month == Unicode::CalendarPatternStyle::Short) {
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
date_time_format_index = date_time_formats.medium_format;
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
} else {
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
date_time_format_index = date_time_formats.short_format;
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
}
for (auto const& time_format : time_formats) {
auto format = locale_data.unique_patterns.get(date_time_format_index);
if (time_format.pattern12_index != 0)
format.pattern12_index = replace_pattern(format.pattern_index, time_format.pattern12_index, date_format.pattern_index);
format.pattern_index = replace_pattern(format.pattern_index, time_format.pattern_index, date_format.pattern_index);
LibJS+LibUnicode: Supply field type in CalendarPattern's for-each method Some callers will want different behavior depending on what field is being provided to the callback.
2021-12-07 08:40:06 -05:00
format.for_each_calendar_field_zipped_with(date_format, [](auto& field, auto const& date_field, auto) {
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
if (date_field.has_value())
field = date_field;
});
LibJS+LibUnicode: Supply field type in CalendarPattern's for-each method Some callers will want different behavior depending on what field is being provided to the callback.
2021-12-07 08:40:06 -05:00
format.for_each_calendar_field_zipped_with(time_format, [](auto& field, auto const& time_field, auto) {
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
if (time_field.has_value())
field = time_field;
});
append_if_unique(move(format));
}
}
}
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
static void parse_calendar_symbols(Calendar& calendar, JsonObject const& calendar_object, UnicodeLocaleData& locale_data)
{
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
auto create_symbol_lists = [](size_t size) {
SymbolList narrow_symbol_list;
SymbolList short_symbol_list;
SymbolList long_symbol_list;
narrow_symbol_list.resize(size);
short_symbol_list.resize(size);
long_symbol_list.resize(size);
return Array<SymbolList, 3> { {
move(narrow_symbol_list),
move(short_symbol_list),
move(long_symbol_list),
} };
};
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
CalendarSymbolsList symbols_list {};
LibUnicode: Generate CalendarSymbols in a predetermined order Similar to commit 2a7f36b392, this change moves the generated CalendarSymbol enumeration to the public LibUnicode/NumberFormat.h header with a pre-defined set of symbols that we need. This is to prepare for uniquely generating the CalendarSymbols structure.
2021-12-11 15:31:33 -05:00
auto store_symbol_lists = [&](auto symbol, auto symbol_lists) {
auto symbol_index = to_underlying(symbol);
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
if (symbol_index >= symbols_list.size())
symbols_list.resize(symbol_index + 1);
LibUnicode: Generate CalendarSymbols in a predetermined order Similar to commit 2a7f36b392, this change moves the generated CalendarSymbol enumeration to the public LibUnicode/NumberFormat.h header with a pre-defined set of symbols that we need. This is to prepare for uniquely generating the CalendarSymbols structure.
2021-12-11 15:31:33 -05:00
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
CalendarSymbols symbols {};
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
symbols.narrow_symbols = locale_data.unique_symbol_lists.ensure(move(symbol_lists[0]));
symbols.short_symbols = locale_data.unique_symbol_lists.ensure(move(symbol_lists[1]));
symbols.long_symbols = locale_data.unique_symbol_lists.ensure(move(symbol_lists[2]));
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
auto calendar_symbols_index = locale_data.unique_calendar_symbols.ensure(move(symbols));
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
symbols_list[symbol_index] = calendar_symbols_index;
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
auto parse_era_symbols = [&](auto const& symbols_object) {
auto const& narrow_symbols = symbols_object.get("eraNarrow"sv).as_object();
auto const& short_symbols = symbols_object.get("eraAbbr"sv).as_object();
auto const& long_symbols = symbols_object.get("eraNames"sv).as_object();
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
auto symbol_lists = create_symbol_lists(2);
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
auto append_symbol = [&](auto& symbols, auto const& key, auto symbol) {
if (auto key_index = key.to_uint(); key_index.has_value())
symbols[*key_index] = locale_data.unique_strings.ensure(move(symbol));
};
narrow_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[0], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
short_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[1], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
long_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[2], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
LibUnicode: Generate CalendarSymbols in a predetermined order Similar to commit 2a7f36b392, this change moves the generated CalendarSymbol enumeration to the public LibUnicode/NumberFormat.h header with a pre-defined set of symbols that we need. This is to prepare for uniquely generating the CalendarSymbols structure.
2021-12-11 15:31:33 -05:00
store_symbol_lists(Unicode::CalendarSymbol::Era, move(symbol_lists));
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
auto parse_month_symbols = [&](auto const& symbols_object) {
auto const& narrow_symbols = symbols_object.get("narrow"sv).as_object();
auto const& short_symbols = symbols_object.get("abbreviated"sv).as_object();
auto const& long_symbols = symbols_object.get("wide"sv).as_object();
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
auto symbol_lists = create_symbol_lists(12);
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
auto append_symbol = [&](auto& symbols, auto const& key, auto symbol) {
auto key_index = key.to_uint().value() - 1;
symbols[key_index] = locale_data.unique_strings.ensure(move(symbol));
};
narrow_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[0], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
short_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[1], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
long_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[2], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
LibUnicode: Generate CalendarSymbols in a predetermined order Similar to commit 2a7f36b392, this change moves the generated CalendarSymbol enumeration to the public LibUnicode/NumberFormat.h header with a pre-defined set of symbols that we need. This is to prepare for uniquely generating the CalendarSymbols structure.
2021-12-11 15:31:33 -05:00
store_symbol_lists(Unicode::CalendarSymbol::Month, move(symbol_lists));
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
auto parse_weekday_symbols = [&](auto const& symbols_object) {
auto const& narrow_symbols = symbols_object.get("narrow"sv).as_object();
auto const& short_symbols = symbols_object.get("abbreviated"sv).as_object();
auto const& long_symbols = symbols_object.get("wide"sv).as_object();
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
auto symbol_lists = create_symbol_lists(7);
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
auto append_symbol = [&](auto& symbols, auto const& key, auto symbol) {
if (key == "sun"sv)
symbols[to_underlying(Unicode::Weekday::Sunday)] = locale_data.unique_strings.ensure(move(symbol));
else if (key == "mon"sv)
symbols[to_underlying(Unicode::Weekday::Monday)] = locale_data.unique_strings.ensure(move(symbol));
else if (key == "tue"sv)
symbols[to_underlying(Unicode::Weekday::Tuesday)] = locale_data.unique_strings.ensure(move(symbol));
else if (key == "wed"sv)
symbols[to_underlying(Unicode::Weekday::Wednesday)] = locale_data.unique_strings.ensure(move(symbol));
else if (key == "thu"sv)
symbols[to_underlying(Unicode::Weekday::Thursday)] = locale_data.unique_strings.ensure(move(symbol));
else if (key == "fri"sv)
symbols[to_underlying(Unicode::Weekday::Friday)] = locale_data.unique_strings.ensure(move(symbol));
else if (key == "sat"sv)
symbols[to_underlying(Unicode::Weekday::Saturday)] = locale_data.unique_strings.ensure(move(symbol));
};
narrow_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[0], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
short_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[1], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
long_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[2], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
LibUnicode: Generate CalendarSymbols in a predetermined order Similar to commit 2a7f36b392, this change moves the generated CalendarSymbol enumeration to the public LibUnicode/NumberFormat.h header with a pre-defined set of symbols that we need. This is to prepare for uniquely generating the CalendarSymbols structure.
2021-12-11 15:31:33 -05:00
store_symbol_lists(Unicode::CalendarSymbol::Weekday, move(symbol_lists));
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
auto parse_day_period_symbols = [&](auto const& symbols_object) {
auto const& narrow_symbols = symbols_object.get("narrow"sv).as_object();
auto const& short_symbols = symbols_object.get("abbreviated"sv).as_object();
auto const& long_symbols = symbols_object.get("wide"sv).as_object();
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
auto symbol_lists = create_symbol_lists(10);
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
auto append_symbol = [&](auto& symbols, auto const& key, auto symbol) {
LibUnicode: Parse and generate secondary day period symbols Generate morning2, afternoon2, evening2, and night2 symbols.
2021-12-10 11:59:58 -05:00
if (auto day_period = day_period_from_string(key); day_period.has_value())
symbols[to_underlying(*day_period)] = locale_data.unique_strings.ensure(move(symbol));
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
narrow_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[0], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
short_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[1], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
long_symbols.for_each_member([&](auto const& key, JsonValue const& value) {
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
append_symbol(symbol_lists[2], key, value.as_string());
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
});
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
LibUnicode: Generate CalendarSymbols in a predetermined order Similar to commit 2a7f36b392, this change moves the generated CalendarSymbol enumeration to the public LibUnicode/NumberFormat.h header with a pre-defined set of symbols that we need. This is to prepare for uniquely generating the CalendarSymbols structure.
2021-12-11 15:31:33 -05:00
store_symbol_lists(Unicode::CalendarSymbol::DayPeriod, move(symbol_lists));
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
};
parse_era_symbols(calendar_object.get("eras"sv).as_object());
parse_month_symbols(calendar_object.get("months"sv).as_object().get("format"sv).as_object());
parse_weekday_symbols(calendar_object.get("days"sv).as_object().get("format"sv).as_object());
parse_day_period_symbols(calendar_object.get("dayPeriods"sv).as_object().get("format"sv).as_object());
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
calendar.symbols = locale_data.unique_calendar_symbols_lists.ensure(move(symbols_list));
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
}
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
static ErrorOr<void> parse_calendars(String locale_calendars_path, UnicodeLocaleData& locale_data, Locale& locale)
{
LexicalPath calendars_path(move(locale_calendars_path));
if (!calendars_path.basename().starts_with("ca-"sv))
return {};
auto calendars_file = TRY(Core::File::open(calendars_path.string(), Core::OpenMode::ReadOnly));
auto calendars = TRY(JsonValue::from_string(calendars_file->read_all()));
auto const& main_object = calendars.as_object().get("main"sv);
auto const& locale_object = main_object.as_object().get(calendars_path.parent().basename());
auto const& dates_object = locale_object.as_object().get("dates"sv);
auto const& calendars_object = dates_object.as_object().get("calendars"sv);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
auto parse_patterns = [&](auto const& patterns_object, auto const& skeletons_object, Vector<CalendarPattern>* patterns) {
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
auto parse_pattern = [&](auto name) {
auto format = patterns_object.get(name);
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
auto skeleton = skeletons_object.get(name);
auto format_index = parse_date_time_pattern(format.as_string(), skeleton.as_string_or(String::empty()), locale_data).value();
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
if (patterns)
patterns->append(locale_data.unique_patterns.get(format_index));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
return format_index;
};
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
CalendarFormat formats {};
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
formats.full_format = parse_pattern("full"sv);
formats.long_format = parse_pattern("long"sv);
formats.medium_format = parse_pattern("medium"sv);
formats.short_format = parse_pattern("short"sv);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
return locale_data.unique_formats.ensure(move(formats));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
calendars_object.as_object().for_each_member([&](auto const& calendar_name, JsonValue const& value) {
LibUnicode: Do not generate data for "generic" calendars This is not a calendar supported by ECMA-402, so let's not waste space with its data. Further, don't generate "gregorian" as a valid Unicode locale extension keyword. It's an invalid type identifier, thus cannot be used in locales such as "en-u-ca-gregorian".
2021-11-30 21:23:13 -05:00
// The generic calendar is not a supported Unicode calendar key, so skip it:
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Locale/calendar#unicode_calendar_keys
if (calendar_name == "generic"sv)
return;
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
Calendar calendar {};
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
CalendarPatternList available_formats {};
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
if (!locale_data.calendars.contains_slow(calendar_name))
locale_data.calendars.append(calendar_name);
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
Vector<CalendarPattern> date_formats;
Vector<CalendarPattern> time_formats;
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
auto const& date_formats_object = value.as_object().get("dateFormats"sv);
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
auto const& date_skeletons_object = value.as_object().get("dateSkeletons"sv);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
calendar.date_formats = parse_patterns(date_formats_object.as_object(), date_skeletons_object.as_object(), &date_formats);
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
auto const& time_formats_object = value.as_object().get("timeFormats"sv);
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
auto const& time_skeletons_object = value.as_object().get("timeSkeletons"sv);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
calendar.time_formats = parse_patterns(time_formats_object.as_object(), time_skeletons_object.as_object(), &time_formats);
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
auto const& date_time_formats_object = value.as_object().get("dateTimeFormats"sv);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
calendar.date_time_formats = parse_patterns(date_time_formats_object.as_object(), JsonObject {}, nullptr);
LibUnicode: Parse and generate available candidate format patterns These formats are used by ECMA-402 when neither a date nor time style is specified. In that case, these patterns are searched for a best match.
2021-11-27 17:31:31 -05:00
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
auto const& available_formats_object = date_time_formats_object.as_object().get("availableFormats"sv);
available_formats_object.as_object().for_each_member([&](auto const& skeleton, JsonValue const& pattern) {
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
auto pattern_index = parse_date_time_pattern(pattern.as_string(), skeleton, locale_data);
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
if (!pattern_index.has_value())
return;
auto const& format = locale_data.unique_patterns.get(*pattern_index);
if (format.contains_only_date_fields())
date_formats.append(format);
else if (format.contains_only_time_fields())
time_formats.append(format);
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
if (!available_formats.contains_slow(*pattern_index))
available_formats.append(*pattern_index);
LibUnicode: Parse and generate available candidate format patterns These formats are used by ECMA-402 when neither a date nor time style is specified. In that case, these patterns are searched for a best match.
2021-11-27 17:31:31 -05:00
});
LibUnicode: Generate missing format patterns TR-35 describes how to combine date, time, and available formats with date-time format patterns to generate more available format patterns: https://unicode.org/reports/tr35/tr35-dates.html#Missing_Skeleton_Fields Use these steps to generate ~400 new patterns for each calendar. These are required for ECMA-402's BasicFormatMatcher to produce reasonable results.
2021-12-01 13:20:25 -05:00
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
auto const& interval_formats_object = date_time_formats_object.as_object().get("intervalFormats"sv);
parse_interval_patterns(calendar, interval_formats_object.as_object(), locale_data);
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
generate_missing_patterns(calendar, available_formats, move(date_formats), move(time_formats), locale_data);
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
parse_calendar_symbols(calendar, value.as_object(), locale_data);
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
calendar.available_formats = locale_data.unique_pattern_lists.ensure(move(available_formats));
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
locale.calendars.set(calendar_name, locale_data.unique_calendars.ensure(move(calendar)));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
});
return {};
}
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
static ErrorOr<void> parse_time_zone_names(String locale_time_zone_names_path, UnicodeLocaleData& locale_data, Locale& locale)
{
LexicalPath time_zone_names_path(move(locale_time_zone_names_path));
time_zone_names_path = time_zone_names_path.append("timeZoneNames.json"sv);
auto time_zone_names_file = TRY(Core::File::open(time_zone_names_path.string(), Core::OpenMode::ReadOnly));
auto time_zone_names = TRY(JsonValue::from_string(time_zone_names_file->read_all()));
auto const& main_object = time_zone_names.as_object().get("main"sv);
auto const& locale_object = main_object.as_object().get(time_zone_names_path.parent().basename());
auto const& dates_object = locale_object.as_object().get("dates"sv);
auto const& time_zone_names_object = dates_object.as_object().get("timeZoneNames"sv);
auto const& meta_zone_object = time_zone_names_object.as_object().get("metazone"sv);
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
auto const& gmt_zero_format_string = time_zone_names_object.as_object().get("gmtZeroFormat"sv);
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
if (meta_zone_object.is_null())
return {};
auto parse_name = [&](StringView type, JsonObject const& meta_zone_object) -> Optional<StringIndexType> {
auto const& names = meta_zone_object.get(type);
if (!names.is_object())
return {};
auto const& daylight = names.as_object().get("daylight"sv);
if (daylight.is_string())
return locale_data.unique_strings.ensure(daylight.as_string());
auto const& standard = names.as_object().get("standard"sv);
if (standard.is_string())
return locale_data.unique_strings.ensure(standard.as_string());
return {};
};
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
TimeZoneList time_zones;
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
TimeZoneFormat time_zone_formats {};
time_zone_formats.gmt_zero_format = locale_data.unique_strings.ensure(gmt_zero_format_string.as_string());
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
auto parse_time_zone = [&](StringView meta_zone, JsonObject const& meta_zone_object) {
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
auto const& golden_zones = locale_data.meta_zones.find(meta_zone)->value;
LibUnicode: Remove unused fields from generated structures A couple of structures held a string index that is unused. Removing them also removes the string values from the unique string list.
2021-12-13 08:16:24 -05:00
TimeZone time_zone {};
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
if (auto long_name = parse_name("long"sv, meta_zone_object); long_name.has_value())
time_zone.long_name = long_name.value();
if (auto short_name = parse_name("short"sv, meta_zone_object); short_name.has_value())
time_zone.short_name = short_name.value();
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
for (auto const& golden_zone : golden_zones) {
auto time_zone_index = locale_data.time_zones.find_first_index(golden_zone).value();
time_zones[time_zone_index] = locale_data.unique_time_zones.ensure(move(time_zone));
}
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
};
meta_zone_object.as_object().for_each_member([&](auto const& meta_zone, JsonValue const&) {
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
auto const& golden_zones = locale_data.meta_zones.find(meta_zone)->value;
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
LibUnicode: Do not assume time zones & meta zones have a 1-to-1 mapping The generator parses metaZones.json to form a mapping of meta zones to time zones (AKA "golden zone" in TR-35). This parser errantly assumed this was a 1-to-1 mapping.
2022-01-06 10:40:18 -05:00
for (auto const& golden_zone : golden_zones) {
if (!locale_data.time_zones.contains_slow(golden_zone))
locale_data.time_zones.append(golden_zone);
}
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
});
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
time_zones.resize(locale_data.time_zones.size());
quick_sort(locale_data.time_zones);
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
meta_zone_object.as_object().for_each_member([&](auto const& meta_zone, JsonValue const& value) {
parse_time_zone(meta_zone, value.as_object());
});
// The long and short names for UTC are not under the "timeZoneNames/metazone" object, but are under "timeZoneNames/zone/Etc".
auto const& zone_object = time_zone_names_object.as_object().get("zone"sv);
auto const& etc_object = zone_object.as_object().get("Etc"sv);
auto const& utc_object = etc_object.as_object().get("UTC"sv);
parse_time_zone("UTC"sv, utc_object.as_object());
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
locale.time_zones = locale_data.unique_time_zone_lists.ensure(move(time_zones));
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
locale.time_zone_formats = locale_data.unique_time_zone_formats.ensure(move(time_zone_formats));
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
return {};
}
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
static ErrorOr<void> parse_day_periods(String core_path, UnicodeLocaleData& locale_data)
{
// https://unicode.org/reports/tr35/tr35-dates.html#Day_Period_Rule_Sets
LexicalPath day_periods_path(move(core_path));
day_periods_path = day_periods_path.append("supplemental"sv);
day_periods_path = day_periods_path.append("dayPeriods.json"sv);
auto day_periods_file = TRY(Core::File::open(day_periods_path.string(), Core::OpenMode::ReadOnly));
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
auto locale_day_periods = TRY(JsonValue::from_string(day_periods_file->read_all()));
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
auto const& supplemental_object = locale_day_periods.as_object().get("supplemental"sv);
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
auto const& day_periods_object = supplemental_object.as_object().get("dayPeriodRuleSet"sv);
auto parse_hour = [](auto const& time) {
auto hour_end_index = time.find(':').value();
// The times are in the form "hours:minutes", but we only need the hour segment.
// TR-35 explicitly states that minutes other than :00 must not be used.
VERIFY(time.substring_view(hour_end_index) == ":00"sv);
auto hour = time.substring_view(0, hour_end_index);
return hour.template to_uint<u8>().value();
};
auto parse_day_period = [&](auto const& symbol, auto const& ranges) -> Optional<DayPeriod> {
auto day_period = day_period_from_string(symbol);
if (!day_period.has_value())
return {};
auto begin = parse_hour(ranges.get("_from"sv).as_string());
auto end = parse_hour(ranges.get("_before"sv).as_string());
return DayPeriod { *day_period, begin, end };
};
day_periods_object.as_object().for_each_member([&](auto const& language, JsonValue const& value) {
auto locale = locale_data.locales.find(language);
if (locale == locale_data.locales.end())
return;
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
DayPeriodList day_periods;
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
value.as_object().for_each_member([&](auto const& symbol, JsonValue const& ranges) {
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
if (auto day_period = parse_day_period(symbol, ranges.as_object()); day_period.has_value()) {
auto day_period_index = locale_data.unique_day_periods.ensure(day_period.release_value());
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
day_periods.append(day_period_index);
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
}
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
});
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
locale->value.day_periods = locale_data.unique_day_period_lists.ensure(move(day_periods));
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
});
return {};
};
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
static ErrorOr<void> parse_all_locales(String core_path, String dates_path, UnicodeLocaleData& locale_data)
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
{
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
TRY(parse_hour_cycles(core_path, locale_data));
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
TRY(parse_meta_zones(core_path, locale_data));
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
auto dates_iterator = TRY(path_to_dir_iterator(move(dates_path)));
auto remove_variants_from_path = [&](String path) -> ErrorOr<String> {
auto parsed_locale = TRY(CanonicalLanguageID<StringIndexType>::parse(locale_data.unique_strings, LexicalPath::basename(path)));
StringBuilder builder;
builder.append(locale_data.unique_strings.get(parsed_locale.language));
if (auto script = locale_data.unique_strings.get(parsed_locale.script); !script.is_empty())
builder.appendff("-{}", script);
if (auto region = locale_data.unique_strings.get(parsed_locale.region); !region.is_empty())
builder.appendff("-{}", region);
return builder.build();
};
while (dates_iterator.has_next()) {
auto dates_path = TRY(next_path_from_dir_iterator(dates_iterator));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
auto calendars_iterator = TRY(path_to_dir_iterator(dates_path, {}));
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
auto language = TRY(remove_variants_from_path(dates_path));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
auto& locale = locale_data.locales.ensure(language);
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
while (calendars_iterator.has_next()) {
auto calendars_path = TRY(next_path_from_dir_iterator(calendars_iterator));
TRY(parse_calendars(move(calendars_path), locale_data, locale));
}
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
TRY(parse_time_zone_names(move(dates_path), locale_data, locale));
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
}
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
TRY(parse_day_periods(move(core_path), locale_data));
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
return {};
}
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
static String format_identifier(StringView owner, String identifier)
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
{
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
identifier = identifier.replace("-"sv, "_"sv, true);
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
identifier = identifier.replace("/"sv, "_"sv, true);
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
if (all_of(identifier, is_ascii_digit))
return String::formatted("{}_{}", owner[0], identifier);
if (is_ascii_lower_alpha(identifier[0]))
return String::formatted("{:c}{}", to_ascii_uppercase(identifier[0]), identifier.substring_view(1));
return identifier;
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
}
static void generate_unicode_locale_header(Core::File& file, UnicodeLocaleData& locale_data)
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
{
StringBuilder builder;
SourceGenerator generator { builder };
generator.append(R"~~~(
#pragma once
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
#include <AK/Types.h>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
namespace Unicode {
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
)~~~");
LibUnicode: Hard-code an alias from the Gregorian calendar to Gregory This alias exists because the name "Gregorian" is too long to be used in a locale identifier, i.e. "en-u-ca-gregorian" is invalid. Aliases for calendars are defined here: https://github.com/unicode-org/cldr-json/blob/main/cldr-json/cldr-bcp47/bcp47/calendar.json However, CLDR version 40 neglected to actually include the cldr-bcp47 package in its release, so we don't have access to this data. So for now hard-code this alias so that JavaScript can actually access it. See: https://unicode-org.atlassian.net/browse/CLDR-15158
2021-11-28 09:25:15 -05:00
generate_enum(generator, format_identifier, "Calendar"sv, {}, locale_data.calendars, locale_data.calendar_aliases);
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
generate_enum(generator, format_identifier, "HourCycleRegion"sv, {}, locale_data.hour_cycle_regions);
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
generate_enum(generator, format_identifier, "TimeZone"sv, {}, locale_data.time_zones);
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
generator.append(R"~~~(
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
}
)~~~");
VERIFY(file.write(generator.as_string_view()));
}
static void generate_unicode_locale_implementation(Core::File& file, UnicodeLocaleData& locale_data)
{
StringBuilder builder;
SourceGenerator generator { builder };
generator.set("string_index_type"sv, s_string_index_type);
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
generator.set("calendar_pattern_index_type"sv, s_calendar_pattern_index_type);
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
generator.set("calendar_pattern_list_index_type"sv, s_calendar_pattern_list_index_type);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
generator.set("calendar_range_pattern_index_type"sv, s_calendar_range_pattern_index_type);
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
generator.set("calendar_range_pattern_list_index_type"sv, s_calendar_range_pattern_list_index_type);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
generator.set("calendar_format_index_type"sv, s_calendar_format_index_type);
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
generator.set("symbol_list_index_type"sv, s_symbol_list_index_type);
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
generator.set("calendar_symbols_index_type"sv, s_calendar_symbols_index_type);
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
generator.set("calendar_symbols_list_index_type"sv, s_calendar_symbols_list_index_type);
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
generator.set("calendar_index_type"sv, s_calendar_index_type);
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
generator.set("time_zone_index_type"sv, s_time_zone_index_type);
Lagom: Bind `time_zone_list_index_type` in the generator The variable `s_time_zone_list_index_type` seems to be unused (detected when compiling with clang), and it seems logical to bind it even it if it is not used for now.
2021-12-16 15:42:43 +01:00
generator.set("time_zone_list_index_type"sv, s_time_zone_list_index_type);
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
generator.set("day_period_index_type"sv, s_day_period_index_type);
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
generator.set("day_period_list_index_type"sv, s_day_period_list_index_type);
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
generator.append(R"~~~(
#include <AK/Array.h>
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
#include <AK/BinarySearch.h>
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
#include <AK/Optional.h>
#include <AK/StringView.h>
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
#include <LibUnicode/DateTimeFormat.h>
#include <LibUnicode/Locale.h>
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
#include <LibUnicode/UnicodeDateTimeFormat.h>
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
namespace Unicode {
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
)~~~");
locale_data.unique_strings.generate(generator);
generator.append(R"~~~(
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
template <typename SourceType, typename TargetType>
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
static void convert_calendar_fields(SourceType const& source, TargetType& target)
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
{
if (source.era != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.era = static_cast<CalendarPatternStyle>(source.era);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.year != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.year = static_cast<CalendarPatternStyle>(source.year);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.month != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.month = static_cast<CalendarPatternStyle>(source.month);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.weekday != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.weekday = static_cast<CalendarPatternStyle>(source.weekday);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.day != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.day = static_cast<CalendarPatternStyle>(source.day);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.day_period != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.day_period = static_cast<CalendarPatternStyle>(source.day_period);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.hour != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.hour = static_cast<CalendarPatternStyle>(source.hour);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.minute != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.minute = static_cast<CalendarPatternStyle>(source.minute);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.second != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.second = static_cast<CalendarPatternStyle>(source.second);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (source.fractional_second_digits != -1)
target.fractional_second_digits = static_cast<u8>(source.fractional_second_digits);
if (source.time_zone_name != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
target.time_zone_name = static_cast<CalendarPatternStyle>(source.time_zone_name);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
struct CalendarPatternImpl {
CalendarPattern to_unicode_calendar_pattern() const {
CalendarPattern calendar_pattern {};
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
calendar_pattern.skeleton = s_string_list[skeleton];
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
calendar_pattern.pattern = s_string_list[pattern];
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
if (pattern12 != 0)
calendar_pattern.pattern12 = s_string_list[pattern12];
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
convert_calendar_fields(*this, calendar_pattern);
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
return calendar_pattern;
}
LibUnicode: Parse and generate format pattern skeletons Pattern skeletons are more or less the "key" of format patterns. Every format pattern is assigned a skeleton. Interval patterns (which are not yet parsed) are also assigned a skeleton - this is used to match them to an "owning" format pattern. So we will use the skeleton generated here to match format patterns at runtime with their available interval patterns. An alternative approach would be to append interval patterns directly to their owning format pattern, but this has some draw backs: 1. Skeletons aren't totally unique. A skeleton may appear in both the "dateFormats" and "availableFormats" objects, in which case the same interval formats would be generated more than once. 2. Otherwise unique format patterns may only differ by the interval patterns assigned to them. This would cause the UniqueStorage for the format patterns to increase in size, impacting both compile times and libunicode.so size.
2021-12-08 13:42:04 -05:00
@string_index_type@ skeleton { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
@string_index_type@ pattern { 0 };
LibUnicode: Fully parse date-time formatting patterns Similar to NumberFormat, replace the segments of date-time patterns with partitions that can be split at runtime. Also generate the pattern style fields for e.g. era, day, hour, etc.
2021-11-30 08:59:19 -05:00
@string_index_type@ pattern12 { 0 };
i8 era { -1 };
i8 year { -1 };
i8 month { -1 };
i8 weekday { -1 };
i8 day { -1 };
i8 day_period { -1 };
i8 hour { -1 };
i8 minute { -1 };
i8 second { -1 };
i8 fractional_second_digits { -1 };
i8 time_zone_name { -1 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
struct CalendarRangePatternImpl {
CalendarRangePattern to_unicode_calendar_range_pattern() const {
CalendarRangePattern calendar_range_pattern {};
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (field != -1)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
calendar_range_pattern.field = static_cast<CalendarRangePattern::Field>(field);
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
calendar_range_pattern.start_range = s_string_list[start_range];
calendar_range_pattern.separator = s_string_list[separator];
calendar_range_pattern.end_range = s_string_list[end_range];
convert_calendar_fields(*this, calendar_range_pattern);
return calendar_range_pattern;
}
@string_index_type@ skeleton { 0 };
i8 field { -1 };
@string_index_type@ start_range { 0 };
@string_index_type@ separator { 0 };
@string_index_type@ end_range { 0 };
i8 era { -1 };
i8 year { -1 };
i8 month { -1 };
i8 weekday { -1 };
i8 day { -1 };
i8 day_period { -1 };
i8 hour { -1 };
i8 minute { -1 };
i8 second { -1 };
i8 fractional_second_digits { -1 };
i8 time_zone_name { -1 };
};
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
)~~~");
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
locale_data.unique_patterns.generate(generator, "CalendarPatternImpl"sv, "s_calendar_patterns"sv, 10);
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
locale_data.unique_pattern_lists.generate(generator, s_calendar_pattern_index_type, "s_calendar_pattern_lists"sv);
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
locale_data.unique_range_patterns.generate(generator, "CalendarRangePatternImpl"sv, "s_calendar_range_patterns"sv, 10);
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
locale_data.unique_range_pattern_lists.generate(generator, s_calendar_range_pattern_index_type, "s_calendar_range_pattern_lists"sv);
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
generator.append(R"~~~(
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
struct CalendarFormatImpl {
CalendarFormat to_unicode_calendar_format() const {
CalendarFormat calendar_format {};
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
calendar_format.full_format = s_calendar_patterns[full_format].to_unicode_calendar_pattern();
calendar_format.long_format = s_calendar_patterns[long_format].to_unicode_calendar_pattern();
calendar_format.medium_format = s_calendar_patterns[medium_format].to_unicode_calendar_pattern();
calendar_format.short_format = s_calendar_patterns[short_format].to_unicode_calendar_pattern();
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
return calendar_format;
}
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
@calendar_pattern_index_type@ full_format { 0 };
@calendar_pattern_index_type@ long_format { 0 };
@calendar_pattern_index_type@ medium_format { 0 };
@calendar_pattern_index_type@ short_format { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
struct CalendarSymbols {
@symbol_list_index_type@ narrow_symbols { 0 };
@symbol_list_index_type@ short_symbols { 0 };
@symbol_list_index_type@ long_symbols { 0 };
};
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
struct CalendarData {
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
@calendar_format_index_type@ date_formats { 0 };
@calendar_format_index_type@ time_formats { 0 };
@calendar_format_index_type@ date_time_formats { 0 };
@calendar_pattern_list_index_type@ available_formats { 0 };
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
@calendar_range_pattern_index_type@ default_range_format { 0 };
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
@calendar_range_pattern_list_index_type@ range_formats { 0 };
@calendar_range_pattern_list_index_type@ range12_formats { 0 };
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
@calendar_symbols_list_index_type@ symbols { 0 };
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
struct TimeZoneData {
@string_index_type@ long_name { 0 };
@string_index_type@ short_name { 0 };
};
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
struct TimeZoneFormat {
@string_index_type@ gmt_zero_format { 0 };
};
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
struct DayPeriodData {
u8 day_period { 0 };
u8 begin { 0 };
u8 end { 0 };
};
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
)~~~");
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
locale_data.unique_formats.generate(generator, "CalendarFormatImpl"sv, "s_calendar_formats"sv, 10);
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
locale_data.unique_symbol_lists.generate(generator, s_string_index_type, "s_symbol_lists"sv);
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
locale_data.unique_calendar_symbols.generate(generator, "CalendarSymbols"sv, "s_calendar_symbols"sv, 10);
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
locale_data.unique_calendar_symbols_lists.generate(generator, s_calendar_symbols_index_type, "s_calendar_symbol_lists"sv);
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
locale_data.unique_calendars.generate(generator, "CalendarData"sv, "s_calendars"sv, 10);
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
locale_data.unique_time_zones.generate(generator, "TimeZoneData"sv, "s_time_zones"sv, 30);
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
locale_data.unique_time_zone_lists.generate(generator, s_time_zone_index_type, "s_time_zone_lists"sv);
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
locale_data.unique_time_zone_formats.generate(generator, "TimeZoneFormat"sv, "s_time_zone_formats"sv, 30);
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
locale_data.unique_day_periods.generate(generator, "DayPeriodData"sv, "s_day_periods"sv, 30);
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
locale_data.unique_day_period_lists.generate(generator, s_day_period_index_type, "s_day_period_lists"sv);
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
locale_data.unique_hour_cycle_lists.generate(generator, "u8"sv, "s_hour_cycle_lists"sv);
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
auto append_calendars = [&](String name, auto const& calendars) {
generator.set("name", name);
generator.set("size", String::number(calendars.size()));
generator.append(R"~~~(
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
static constexpr Array<@calendar_index_type@, @size@> @name@ { {)~~~");
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
bool first = true;
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
for (auto const& calendar_key : locale_data.calendars) {
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
auto calendar = calendars.find(calendar_key)->value;
generator.append(first ? " " : ", ");
generator.append(String::number(calendar));
first = false;
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
}
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
generator.append(" } };");
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
};
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
auto append_mapping = [&](auto const& keys, auto const& map, auto type, auto name, auto mapping_getter) {
generator.set("type", type);
generator.set("name", name);
generator.set("size", String::number(keys.size()));
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
generator.append(R"~~~(
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
static constexpr Array<@type@, @size@> @name@ { {)~~~");
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
bool first = true;
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
for (auto const& key : keys) {
auto const& value = map.find(key)->value;
auto mapping = mapping_getter(value);
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
generator.append(first ? " " : ", ");
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
generator.append(String::number(mapping));
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
first = false;
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
}
generator.append(" } };");
};
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
auto locales = locale_data.locales.keys();
quick_sort(locales);
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
generate_mapping(generator, locale_data.locales, s_calendar_index_type, "s_locale_calendars"sv, "s_calendars_{}", [&](auto const& name, auto const& value) { append_calendars(name, value.calendars); });
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
append_mapping(locales, locale_data.locales, s_time_zone_index_type, "s_locale_time_zones"sv, [](auto const& locale) { return locale.time_zones; });
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
append_mapping(locales, locale_data.locales, s_time_zone_format_index_type, "s_locale_time_zone_formats"sv, [](auto const& locale) { return locale.time_zone_formats; });
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
append_mapping(locales, locale_data.locales, s_day_period_index_type, "s_locale_day_periods"sv, [](auto const& locale) { return locale.day_periods; });
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
append_mapping(locale_data.hour_cycle_regions, locale_data.hour_cycles, s_hour_cycle_list_index_type, "s_hour_cycles"sv, [](auto const& hour_cycles) { return hour_cycles; });
generator.append("\n");
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
auto append_from_string = [&](StringView enum_title, StringView enum_snake, auto const& values, Vector<Alias> const& aliases = {}) {
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
HashValueMap<String> hashes;
hashes.ensure_capacity(values.size());
for (auto const& value : values)
hashes.set(value.hash(), format_identifier(enum_title, value));
LibUnicode: Hard-code an alias from the Gregorian calendar to Gregory This alias exists because the name "Gregorian" is too long to be used in a locale identifier, i.e. "en-u-ca-gregorian" is invalid. Aliases for calendars are defined here: https://github.com/unicode-org/cldr-json/blob/main/cldr-json/cldr-bcp47/bcp47/calendar.json However, CLDR version 40 neglected to actually include the cldr-bcp47 package in its release, so we don't have access to this data. So for now hard-code this alias so that JavaScript can actually access it. See: https://unicode-org.atlassian.net/browse/CLDR-15158
2021-11-28 09:25:15 -05:00
for (auto const& alias : aliases)
hashes.set(alias.alias.hash(), format_identifier(enum_title, alias.alias));
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
generate_value_from_string(generator, "{}_from_string"sv, enum_title, enum_snake, move(hashes));
};
LibUnicode: Hard-code an alias from the Gregorian calendar to Gregory This alias exists because the name "Gregorian" is too long to be used in a locale identifier, i.e. "en-u-ca-gregorian" is invalid. Aliases for calendars are defined here: https://github.com/unicode-org/cldr-json/blob/main/cldr-json/cldr-bcp47/bcp47/calendar.json However, CLDR version 40 neglected to actually include the cldr-bcp47 package in its release, so we don't have access to this data. So for now hard-code this alias so that JavaScript can actually access it. See: https://unicode-org.atlassian.net/browse/CLDR-15158
2021-11-28 09:25:15 -05:00
append_from_string("Calendar"sv, "calendar"sv, locale_data.calendars, locale_data.calendar_aliases);
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
append_from_string("HourCycleRegion"sv, "hour_cycle_region"sv, locale_data.hour_cycle_regions);
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
append_from_string("TimeZone"sv, "time_zone"sv, locale_data.time_zones);
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
generator.append(R"~~~(
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<HourCycle> get_regional_hour_cycles(StringView region)
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
{
auto region_value = hour_cycle_region_from_string(region);
if (!region_value.has_value())
return {};
auto region_index = to_underlying(*region_value);
LibUnicode: Generate unique lists of hour cycles
2021-12-12 22:13:17 -05:00
auto regional_hour_cycles_index = s_hour_cycles.at(region_index);
auto const& regional_hour_cycles = s_hour_cycle_lists.at(regional_hour_cycles_index);
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<HourCycle> hour_cycles;
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
hour_cycles.ensure_capacity(regional_hour_cycles.size());
for (auto hour_cycle : regional_hour_cycles)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
hour_cycles.unchecked_append(static_cast<HourCycle>(hour_cycle));
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
return hour_cycles;
}
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
static CalendarData const* find_calendar_data(StringView locale, StringView calendar)
{
auto locale_value = locale_from_string(locale);
if (!locale_value.has_value())
return nullptr;
auto calendar_value = calendar_from_string(calendar);
if (!calendar_value.has_value())
return nullptr;
auto locale_index = to_underlying(*locale_value) - 1; // Subtract 1 because 0 == Locale::None.
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
size_t calendar_index = to_underlying(*calendar_value);
auto const& calendar_indices = s_locale_calendars.at(locale_index);
calendar_index = calendar_indices[calendar_index];
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
LibUnicode: Generate unique calendar structures Of the 374 generated calendars, 173 are unique.
2021-12-12 15:32:37 -05:00
return &s_calendars[calendar_index];
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Optional<CalendarFormat> get_calendar_date_format(StringView locale, StringView calendar)
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
{
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
auto const& formats = s_calendar_formats.at(data->date_formats);
return formats.to_unicode_calendar_format();
}
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
return {};
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Optional<CalendarFormat> get_calendar_time_format(StringView locale, StringView calendar)
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
{
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
auto const& formats = s_calendar_formats.at(data->time_formats);
return formats.to_unicode_calendar_format();
}
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
return {};
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Optional<CalendarFormat> get_calendar_date_time_format(StringView locale, StringView calendar)
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
{
LibUnicode: Generate unique calendar format structures There are currently 374 calendars generated, each of which include 3 CalendarFormat structures. Of these 1122 instances, only 167 are unique.
2021-12-11 14:35:13 -05:00
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
auto const& formats = s_calendar_formats.at(data->date_time_formats);
return formats.to_unicode_calendar_format();
}
LibUnicode: Parse and generate date, time, and date-time format patterns
2021-11-27 14:54:48 -05:00
return {};
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<CalendarPattern> get_calendar_available_formats(StringView locale, StringView calendar)
LibUnicode: Parse and generate available candidate format patterns These formats are used by ECMA-402 when neither a date nor time style is specified. In that case, these patterns are searched for a best match.
2021-11-27 17:31:31 -05:00
{
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<CalendarPattern> result {};
LibUnicode: Parse and generate available candidate format patterns These formats are used by ECMA-402 when neither a date nor time style is specified. In that case, these patterns are searched for a best match.
2021-11-27 17:31:31 -05:00
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
auto const& available_formats = s_calendar_pattern_lists.at(data->available_formats);
result.ensure_capacity(available_formats.size());
LibUnicode: Parse and generate available candidate format patterns These formats are used by ECMA-402 when neither a date nor time style is specified. In that case, these patterns are searched for a best match.
2021-11-27 17:31:31 -05:00
LibUnicode: Generate unique lists of calendar patterns Of the 374 generated lists, 152 are unique. These lists have upwards of 1000 entries as well, so the de-duplication is particularly nice.
2021-12-12 14:48:32 -05:00
for (auto const& format : available_formats)
LibUnicode: Generate unique calendar pattern structures Add unique storage for parsed CalendarPattern structures to ensure only one copy of each structure is generated. This doesn't have any impact on libunicode.so with the current generated data. Rather, this prevents the amount of generated data from needlessly growing astronomically once date-time patterns are fully parsed. There will be 173,459 patterns parsed, of which only 22,495 (about 12%) are unique. This change will save a few MB, and will also help compilation times.
2021-12-02 21:16:16 -05:00
result.unchecked_append(s_calendar_patterns[format].to_unicode_calendar_pattern());
LibUnicode: Parse and generate available candidate format patterns These formats are used by ECMA-402 when neither a date nor time style is specified. In that case, these patterns are searched for a best match.
2021-11-27 17:31:31 -05:00
}
return result;
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Optional<CalendarRangePattern> get_calendar_default_range_format(StringView locale, StringView calendar)
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
{
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
auto const& pattern = s_calendar_range_patterns[data->default_range_format];
return pattern.to_unicode_calendar_range_pattern();
}
return {};
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<CalendarRangePattern> get_calendar_range_formats(StringView locale, StringView calendar, StringView skeleton)
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
{
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<CalendarRangePattern> result {};
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
auto const& range_formats = s_calendar_range_pattern_lists.at(data->range_formats);
for (auto format : range_formats) {
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
auto const& pattern = s_calendar_range_patterns[format];
if (skeleton == s_string_list[pattern.skeleton])
result.append(pattern.to_unicode_calendar_range_pattern());
}
}
return result;
}
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<CalendarRangePattern> get_calendar_range12_formats(StringView locale, StringView calendar, StringView skeleton)
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
{
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
Vector<CalendarRangePattern> result {};
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
LibUnicode: Generate unique lists of calendar range patterns Of the 374 range pattern lists and 374 range12 pattern lists, 230 are unique.
2021-12-12 15:05:38 -05:00
auto const& range12_formats = s_calendar_range_pattern_lists.at(data->range12_formats);
for (auto format : range12_formats) {
LibUnicode: Parse and generate date time range format patterns
2021-12-08 17:07:30 -05:00
auto const& pattern = s_calendar_range_patterns[format];
if (skeleton == s_string_list[pattern.skeleton])
result.append(pattern.to_unicode_calendar_range_pattern());
}
}
return result;
}
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
static Span<@string_index_type@ const> find_calendar_symbols(StringView locale, StringView calendar, CalendarSymbol symbol, CalendarPatternStyle style)
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
{
if (auto const* data = find_calendar_data(locale, calendar); data != nullptr) {
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
auto const& symbols_list = s_calendar_symbol_lists[data->symbols];
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
auto symbol_index = to_underlying(symbol);
LibUnicode: Generate unique lists of calendar symbols structures Of the 374 generated lists, 120 are unique.
2021-12-12 14:20:51 -05:00
auto calendar_symbols_index = symbols_list.at(symbol_index);
LibUnicode: Generate unique calendar symbols structures Each of the 374 generated calendars include 4 symbols structures. Of these 1496 structures, only 386 are unique.
2021-12-12 14:04:34 -05:00
auto const& symbols = s_calendar_symbols.at(calendar_symbols_index);
@symbol_list_index_type@ symbol_list_index = 0;
switch (style) {
case CalendarPatternStyle::Narrow:
symbol_list_index = symbols.narrow_symbols;
break;
case CalendarPatternStyle::Short:
symbol_list_index = symbols.short_symbols;
break;
case CalendarPatternStyle::Long:
symbol_list_index = symbols.long_symbols;
break;
default:
VERIFY_NOT_REACHED();
}
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
LibUnicode: Generate unique lists of calendar symbols Each of the 374 generated calendars include 4 sets of symbols, each of which have 3 lists of symbols (narrow, short, long). Of these 4488 lists, only 819 are unique.
2021-12-11 15:15:52 -05:00
return s_symbol_lists.at(symbol_list_index);
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
}
return {};
}
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
Optional<StringView> get_calendar_era_symbol(StringView locale, StringView calendar, CalendarPatternStyle style, Era value)
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
{
auto symbols = find_calendar_symbols(locale, calendar, CalendarSymbol::Era, style);
if (auto value_index = to_underlying(value); value_index < symbols.size())
return s_string_list[symbols.at(value_index)];
return {};
}
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
Optional<StringView> get_calendar_month_symbol(StringView locale, StringView calendar, CalendarPatternStyle style, Month value)
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
{
auto symbols = find_calendar_symbols(locale, calendar, CalendarSymbol::Month, style);
if (auto value_index = to_underlying(value); value_index < symbols.size())
return s_string_list[symbols.at(value_index)];
return {};
}
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
Optional<StringView> get_calendar_weekday_symbol(StringView locale, StringView calendar, CalendarPatternStyle style, Weekday value)
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
{
auto symbols = find_calendar_symbols(locale, calendar, CalendarSymbol::Weekday, style);
if (auto value_index = to_underlying(value); value_index < symbols.size())
return s_string_list[symbols.at(value_index)];
return {};
}
LibUnicode: Dynamically load the generated UnicodeDateTimeFormat symbols
2021-12-15 23:19:17 -05:00
Optional<StringView> get_calendar_day_period_symbol(StringView locale, StringView calendar, CalendarPatternStyle style, DayPeriod value)
LibUnicode: Generate era, month, weekday and day period calendar symbols The parsing in parse_calendar_symbols() might be a bit more verbose than it really needs to be, but it is to ensure the symbols are generated in a known order that we can control with enumerations.
2021-12-06 12:22:39 -05:00
{
auto symbols = find_calendar_symbols(locale, calendar, CalendarSymbol::DayPeriod, style);
if (auto value_index = to_underlying(value); value_index < symbols.size())
return s_string_list[symbols.at(value_index)];
return {};
}
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
Optional<StringView> get_calendar_day_period_symbol_for_hour(StringView locale, StringView calendar, CalendarPatternStyle style, u8 hour)
{
auto locale_value = locale_from_string(locale);
if (!locale_value.has_value())
return {};
auto locale_index = to_underlying(*locale_value) - 1; // Subtract 1 because 0 == Locale::None.
LibUnicode: Generate unique lists of day period structures
2021-12-12 17:45:07 -05:00
auto day_periods_index = s_locale_day_periods[locale_index];
auto day_periods = s_day_period_lists[day_periods_index];
LibUnicode: Generate unique day period structures
2021-12-12 17:11:11 -05:00
for (auto day_period_index : day_periods) {
LibJS+LibUnicode: Handle flexible day periods that roll over midnight When searching for the locale-specific flexible day period for a given hour, we were neglecting to handle cases where the period crosses 00:00. For example, the en locale defines a day period range of [21:00, 06:00). When given the hour of 05:00, we were checking if (21 <= 5 && 5 < 6), thus not recognizing that the hour falls in that period.
2022-01-05 08:24:02 -05:00
auto day_period = s_day_periods[day_period_index];
auto h = hour;
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
LibJS+LibUnicode: Handle flexible day periods that roll over midnight When searching for the locale-specific flexible day period for a given hour, we were neglecting to handle cases where the period crosses 00:00. For example, the en locale defines a day period range of [21:00, 06:00). When given the hour of 05:00, we were checking if (21 <= 5 && 5 < 6), thus not recognizing that the hour falls in that period.
2022-01-05 08:24:02 -05:00
if (day_period.begin > day_period.end) {
day_period.end += 24;
h += 24;
}
if ((day_period.begin <= h) && (h < day_period.end)) {
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
auto period = static_cast<DayPeriod>(day_period.day_period);
return get_calendar_day_period_symbol(locale, calendar, style, period);
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
}
}
// Fallback to fixed periods if the locale does not have flexible day periods.
// TR-35 states that the meaning of AM and PM does not change with locale.
if (hour < 12)
LibUnicode: Convert UnicodeDateTimeFormat to link with weak symbols
2022-01-04 11:29:42 -05:00
return get_calendar_day_period_symbol(locale, calendar, style, DayPeriod::AM);
return get_calendar_day_period_symbol(locale, calendar, style, DayPeriod::PM);
LibUnicode: Parse and generate per-locale day period ranges
2021-12-10 12:57:17 -05:00
}
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
static TimeZoneFormat const* find_time_zone_formats(StringView locale)
{
auto locale_value = locale_from_string(locale);
if (!locale_value.has_value())
return nullptr;
auto locale_index = to_underlying(*locale_value) - 1; // Subtract 1 because 0 == Locale::None.
auto time_zone_format_index = s_locale_time_zone_formats.at(locale_index);
return &s_time_zone_formats.at(time_zone_format_index);
}
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
static TimeZoneData const* find_time_zone_data(StringView locale, StringView time_zone)
{
auto locale_value = locale_from_string(locale);
if (!locale_value.has_value())
return nullptr;
auto time_zone_value = time_zone_from_string(time_zone);
if (!time_zone_value.has_value())
return nullptr;
auto locale_index = to_underlying(*locale_value) - 1; // Subtract 1 because 0 == Locale::None.
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
size_t time_zone_index = to_underlying(*time_zone_value);
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
auto time_zone_list_index = s_locale_time_zones.at(locale_index);
auto const& time_zone_list = s_time_zone_lists.at(time_zone_list_index);
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
LibUnicode: Generate unique lists of time zone structures
2021-12-12 18:21:01 -05:00
time_zone_index = time_zone_list.at(time_zone_index);
LibUnicode: Generate unique time zone structures Each of the 374 locales contain 156 time zone structures. Of these 58,344 structures, 13,578 are unique.
2021-12-12 16:32:28 -05:00
return &s_time_zones[time_zone_index];
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
}
Optional<StringView> get_time_zone_name(StringView locale, StringView time_zone, CalendarPatternStyle style)
{
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
if ((style == CalendarPatternStyle::Long) || (style == CalendarPatternStyle::Short)) {
if (auto const* data = find_time_zone_data(locale, time_zone); data != nullptr) {
auto time_zone_index = style == CalendarPatternStyle::Long ? data->long_name : data->short_name;
if (time_zone_index != 0)
return s_string_list[time_zone_index];
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
}
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
} else {
LibUnicode: Move UTC verification to the scope that requires it In Unicode::get_time_zone_name(), we don't need to require that the time zone is UTC for long- and short-style name lookups. This is required for other styles, because they will depend on TZDB data - so move the VERIFY to that scope.
2022-01-06 10:23:57 -05:00
// FIXME: This becomes more complicated when time zones other than UTC are supported. We will need to know the GMT offset
// of each time zone (which must be parsed from the time zone database, not the CLDR). For now, assuming UTC means
// we can assume a GMT offset of 0, for which the CLDR has a specific format string for the offset styles. Further,
// we will need to parse the "generic" time zone names from timeZoneNames.json.
VERIFY(time_zone == "UTC"sv);
LibUnicode: Add minimal support for generic & offset-based time zones ECMA-402 now supports short-offset, long-offset, short-generic, and long-generic time zone name formatting. For example, in the en-US locale the America/Eastern time zone would be formatted as: short-offset: GMT-5 long-offset: GMT-05:00 short-generic: ET long-generic: Eastern Time We currently only support the UTC time zone, however. Therefore, this very minimal implementation does not consider GMT offset or generic display names. Instead, the CLDR defines specific strings for UTC.
2022-01-02 14:23:24 -05:00
if (auto const* formats = find_time_zone_formats(locale); formats != nullptr)
return s_string_list[formats->gmt_zero_format];
LibUnicode: Parse and generate time zone names in long and short form
2021-12-06 15:46:49 -05:00
}
return {};
}
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
}
)~~~");
VERIFY(file.write(generator.as_string_view()));
}
ErrorOr<int> serenity_main(Main::Arguments arguments)
{
StringView generated_header_path;
StringView generated_implementation_path;
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
StringView core_path;
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
StringView dates_path;
Core::ArgsParser args_parser;
args_parser.add_option(generated_header_path, "Path to the Unicode locale header file to generate", "generated-header-path", 'h', "generated-header-path");
args_parser.add_option(generated_implementation_path, "Path to the Unicode locale implementation file to generate", "generated-implementation-path", 'c', "generated-implementation-path");
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
args_parser.add_option(core_path, "Path to cldr-core directory", "core-path", 'r', "core-path");
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
args_parser.add_option(dates_path, "Path to cldr-dates directory", "dates-path", 'd', "dates-path");
args_parser.parse(arguments);
auto open_file = [&](StringView path) -> ErrorOr<NonnullRefPtr<Core::File>> {
if (path.is_empty()) {
args_parser.print_usage(stderr, arguments.argv[0]);
return Error::from_string_literal("Must provide all command line options"sv);
}
return Core::File::open(path, Core::OpenMode::ReadWrite);
};
auto generated_header_file = TRY(open_file(generated_header_path));
auto generated_implementation_file = TRY(open_file(generated_implementation_path));
UnicodeLocaleData locale_data;
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
TRY(parse_all_locales(core_path, dates_path, locale_data));
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
generate_unicode_locale_header(generated_header_file, locale_data);
generate_unicode_locale_implementation(generated_implementation_file, locale_data);
return 0;
}
Reference in a new issue Copy permalink