2023-11-09 11:27:11 -08:00
{
"compilerOptions" : {
/ * V i s i t h t t p s : //aka.ms/tsconfig to read more about this file */
/ * P r o j e c t s * /
// "incremental": true, /* Save .tsbuildinfo files to allow for incremental compilation of projects. */
// "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */
// "tsBuildInfoFile": "./.tsbuildinfo", /* Specify the path to .tsbuildinfo incremental compilation file. */
// "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects. */
// "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */
// "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */
/ * L a n g u a g e a n d E n v i r o n m e n t * /
2023-11-09 19:11:11 -05:00
"target" : "es2022" / * S e t t h e J a v a S c r i p t l a n g u a g e v e r s i o n f o r e m i t t e d J a v a S c r i p t a n d i n c l u d e c o m p a t i b l e l i b r a r y d e c l a r a t i o n s . * / ,
"lib" : [
"es2022" ,
"dom" ,
"dom.iterable"
] / * S p e c i f y a s e t o f b u n d l e d l i b r a r y d e c l a r a t i o n f i l e s t h a t d e s c r i b e t h e t a r g e t r u n t i m e e n v i r o n m e n t . * / ,
2023-11-09 11:27:11 -08:00
// "jsx": "preserve", /* Specify what JSX code is generated. */
// "experimentalDecorators": true, /* Enable experimental support for TC39 stage 2 draft decorators. */
// "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */
// "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */
// "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
// "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */
// "reactNamespace": "", /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */
// "noLib": true, /* Disable including any library files, including the default lib.d.ts. */
// "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */
// "moduleDetection": "auto", /* Control what method is used to detect module-format JS files. */
/ * M o d u l e s * /
2023-11-09 19:11:11 -05:00
"module" : "NodeNext" / * S p e c i f y w h a t m o d u l e c o d e i s g e n e r a t e d . * / ,
"rootDir" : "./src" / * S p e c i f y t h e r o o t f o l d e r w i t h i n y o u r s o u r c e f i l e s . * / ,
"moduleResolution" : "NodeNext" / * S p e c i f y h o w T y p e S c r i p t l o o k s u p a f i l e f r o m a g i v e n m o d u l e s p e c i f i e r . * / ,
2023-11-09 11:27:11 -08:00
//"baseUrl": "./src", /* Specify the base directory to resolve non-relative module names. */
// "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */
// "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */
// "typeRoots": [], /* Specify multiple folders that act like './node_modules/@types'. */
// "types": [], /* Specify type package names to be included without being referenced in a source file. */
// "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */
// "moduleSuffixes": [], /* List of file name suffixes to search when resolving a module. */
// "resolveJsonModule": true, /* Enable importing .json files. */
// "noResolve": true, /* Disallow 'import's, 'require's or '<reference>'s from expanding the number of files TypeScript should add to a project. */
/ * J a v a S c r i p t S u p p o r t * /
2023-11-09 19:11:11 -05:00
"allowJs" : true / * A l l o w J a v a S c r i p t f i l e s t o b e a p a r t o f y o u r p r o g r a m . U s e t h e ' c h e c k J S ' o p t i o n t o g e t e r r o r s f r o m t h e s e f i l e s . * / ,
"checkJs" : true / * E n a b l e e r r o r r e p o r t i n g i n t y p e - c h e c k e d J a v a S c r i p t f i l e s . * / ,
2023-11-09 11:27:11 -08:00
// "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */
/ * E m i t * /
// "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
// "declarationMap": true, /* Create sourcemaps for d.ts files. */
// "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */
// "sourceMap": true, /* Create source map files for emitted JavaScript files. */
// "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */
2023-11-09 19:11:11 -05:00
"outDir" : "./dist/" / * S p e c i f y a n o u t p u t f o l d e r f o r a l l e m i t t e d f i l e s . * / ,
2023-11-09 11:27:11 -08:00
// "removeComments": true, /* Disable emitting comments. */
// "noEmit": true, /* Disable emitting files from a compilation. */
// "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
// "importsNotUsedAsValues": "remove", /* Specify emit/checking behavior for imports that are only used for types. */
// "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
// "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */
// "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */
// "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */
// "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */
// "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
// "newLine": "crlf", /* Set the newline character for emitting files. */
// "stripInternal": true, /* Disable emitting declarations that have '@internal' in their JSDoc comments. */
// "noEmitHelpers": true, /* Disable generating custom helper functions like '__extends' in compiled output. */
// "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */
// "preserveConstEnums": true, /* Disable erasing 'const enum' declarations in generated code. */
// "declarationDir": "./", /* Specify the output directory for generated declaration files. */
// "preserveValueImports": true, /* Preserve unused imported values in the JavaScript output that would otherwise be removed. */
/ * I n t e r o p C o n s t r a i n t s * /
// "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */
// "allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */
//"esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
// "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
2023-11-09 19:11:11 -05:00
"forceConsistentCasingInFileNames" : true / * E n s u r e t h a t c a s i n g i s c o r r e c t i n i m p o r t s . * / ,
2023-11-09 11:27:11 -08:00
/ * T y p e C h e c k i n g * /
2023-11-09 19:11:11 -05:00
"strict" : true / * E n a b l e a l l s t r i c t t y p e - c h e c k i n g o p t i o n s . * / ,
2023-11-09 11:27:11 -08:00
// "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied 'any' type. */
// "strictNullChecks": true, /* When type checking, take into account 'null' and 'undefined'. */
// "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
// "strictBindCallApply": true, /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */
// "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */
// "noImplicitThis": true, /* Enable error reporting when 'this' is given the type 'any'. */
// "useUnknownInCatchVariables": true, /* Default catch clause variables as 'unknown' instead of 'any'. */
// "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */
// "noUnusedLocals": true, /* Enable error reporting when local variables aren't read. */
// "noUnusedParameters": true, /* Raise an error when a function parameter isn't read. */
// "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */
// "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */
// "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */
// "noUncheckedIndexedAccess": true, /* Add 'undefined' to a type when accessed using an index. */
// "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */
// "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type. */
// "allowUnusedLabels": true, /* Disable error reporting for unused labels. */
// "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */
/ * C o m p l e t e n e s s * /
// "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */
Streaming in-place WACZ creation + CDXJ indexing (#673)
Fixes #674
This PR supersedes #505, and instead of using js-wacz for optimized WACZ
creation:
- generates an 'in-place' or 'streaming' WACZ in the crawler, without
having to copy the data again.
- WACZ contents are streamed to remote upload (or to disk) from existing
files on disk
- CDXJ indices per-WARC are first written to 'warc-cdx' directory, then merged using the linux 'sort' command, and compressed to ZipNum if >50K (or always if using --generateCDX)
- All data in the WARCs is written and read only once
- Should result in significant speed / disk usage improvements:
previously WARC was written once, then read again (for CDXJ indexing),
read again (for adding to new WACZ ZIP), written to disk (into new WACZ
ZIP), read again (if upload to remote endpoint). Now, WARCs are written
once, along with the per-WARC CDXJ, the CDXJ only is reread, sorted and merged on-disk, and all
data is read once to either generate WACZ on disk or upload to remote.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2024-08-29 13:21:20 -07:00
"skipLibCheck" : false / * S k i p t y p e c h e c k i n g a l l . d . t s f i l e s . * /
2023-11-09 11:27:11 -08:00
} ,
2023-11-09 19:11:11 -05:00
"include" : [ "src/**/*" ]
2023-11-09 11:27:11 -08:00
}