^j9 HdZddlZddlZddlZddlZddlmZdZdZdZ dZ dZ ejdjZejd jZ d d ed ed eed edeeeff dZd edeeeefeeefffdZd ededeedeeefdzfdZd ededeedeeefdzfdZdededefdZdedefdZdedeeeeeefefffdZdeeeeeefeffdefdZdeeeeeefeffdedeeeffdZdeededefdZ y)!aOwner of the model-artifacts file format, write side and read side. The model artifacts are the digest-locked file set the trainer produces as one unit: ``models.bin`` (``CMD2``: per-model names and L2 norms, then zlib-compressed dense bigram tables), ``rowmax.bin`` (``CRM1``: per-model row maxima for upper-bound prescreening, digest-locked to the ``models.bin`` they were computed from), ``idf.bin`` (a 65536-byte quantized IDF table), and ``langnorm.bin`` (``CLN1``: per-model norms under the IDF-weighted metric the language tiers score in, digest-locked like ``rowmax.bin``). Keeping both directions in one module means a format change cannot land on one side only: ``scripts/train.py`` calls :func:`write_model_artifacts` and :func:`read_models`, while :mod:`chardet.models` calls :func:`parse_models_bin` and :func:`parse_rowmax_bin` behind its cached accessors. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)PathsCMD2sCRM1$sCLN1z>Iz>ddataoffsetnames chunk_sizereturnc(t|}|dz}tj}i}t}d} |} t|} d} t||krdt|z } |jr|j |j| }nF| | kr)|| | |z}| t|z } |j || }n| sd} |j }|snknj| t|z } ||z }t|dk(r+t|||t|<|jnt|dkDrnt||krt||k(rd}|jr|j |jd}|sM|jsA| | kr<|| | |z}| t|z } |j |d}|s|js| | kr<|s|js| s|j }| t|z } | |k7st||k7rd| d|}t||S) uDecompress the model tables from ``data[offset:]``, one per name. Each model is stored as its own bytes object rather than a memoryview slice of one big blob: mypyc compiles bytes indexing in the scoring hot loop to a native C array access, while memoryview indexing goes through a boxed generic call. Decompression is incremental, one 64 KB table at a time — materializing the whole multi-megabyte blob and slicing it would transiently double the allocation and strand the freed pages in process RSS. Trailing compressed bytes are ignored, as with whole-blob ``zlib.decompress``; the decompressed size is validated instead. :raises ValueError: If the decompressed size is not exactly ``len(names) * 65536``. rFTz&corrupt models.bin: decompressed size z != expected ) lenzlib decompressobj bytearrayunconsumed_tail decompressflushbytescleareof ValueError)rrrr num_models expected_sizedecompmodelstableproducedposendflushedneedpiecechunkextramsgs X/var/www/html/metafoliadorv2/venv/lib/python3.12/site-packages/chardet/models/_format.py_decompress_tablesr(,s&"UJ&M    !F!F KEH C d)CG f+ "s5z!  ! !%%f&<&r?ikeys r'parse_rowmax_binrGs RaL % &- 7 I+c*o.CC C$J/ 3 (1s725G1q5TW-5WX X     s'A4c |ddtk(rc|dt|k(rWt|tt|dzzk(r6t|Dcic]\}}|t |t|dzzd!c}}Sycc}}w)aParse langnorm.bin into per-model language-scoring norms. :param data: Raw bytes of langnorm.bin. :param models_digest: SHA-256 digest of the current models.bin bytes. :param model_keys: Model keys in models.bin header order. :returns: Mapping of model key to its IDF-weighted norm, or ``None`` when the magic, digest, or size does not match -- a stale file would normalize every language score against the wrong length and silently tilt the comparison between languages. Nr*r-r)LANGNORM_MAGICLANGNORM_HEADER_SIZErrCr2rDs r'parse_langnorm_binrKs RaN" ' (M 9 I-J!0CC C$J/ 3 ';a!e'CDQG G     s$A1ridfc|d}tdD]}||}|s |||z||zz }tj|S)aDerive one model's norm under the language-scoring metric. The language tiers compare an input profile weighted by IDF against models weighted by the square root of IDF, so a model's length in that space is the root of the sum of ``weight**2 * idf`` over its bigrams. Scoring folds the model-side factor into the profile and divides by this norm, so the table itself is never rescaled. rr )r0mathsqrt)rrLtotalrEvs r'langnorm_from_tablerRsO E 5\$ !H QUSV^ #E$ 99U r cBtfdtdddDS)zDerive the 256-byte row-maxima table of one dense model table. Entry ``b1`` holds the maximum weight in the model's row for lead byte ``b1``; a row with no bigrams yields 0. c3@K|]}t||dzyw)r+N)max).0startrs r' z$rowmax_from_table..s!SUU553;/0Ssrr r+)rr0)rs`r'rowmax_from_tablerYs SeAuc>RS SSr models_pathc|jsiS|j}|siSt|\}}i}|jD]4\}}i}t dD]}||} | dkDs| ||dz |dzf<|||<6|S)aRead models.bin into the sparse per-model bigram dicts training uses. The inverse of :func:`write_model_artifacts`'s models.bin output: only non-zero weights appear in the dicts. A missing or empty file reads as no models, so a first-ever training run and a full retrain look the same to the caller. :returns: Mapping of model key to ``{(b1, b2): weight}``. :raises ValueError: If the file exists but is corrupt. r rr-)is_file read_bytesr=itemsr0) rZrtables_normsrr:rbigramsidxweights r' read_modelsres      ! ! #D  %d+NFF46F||~ e.0< 9C3ZFz283:./ 9t  Mr rct|}dgdz}|jD]}|D]\}}||dz|zxxdz cc<!|dkDrtj|nd}|dkDrd|z nd}t d}t dD]H} || } | dkDr7tj|| z } t dt| |zdz|| <Dd|| <Jt|S)aACompute the 65536-byte quantized IDF table over all models. For each bigram index the byte holds a scaled inverse document frequency: bigrams present in every model score 1 (minimal signal), bigrams in exactly one model score 255 (maximum signal), and bigrams in no model score 1 (unknown, neutral). rr r-r g?go@g) rvaluesrNlogrr0rUroundr) rrdoc_freqrbb1b2max_idfscale idf_tablercdfidf_vals r' _idf_tablerrsVJsU{H==?* *FB bAg^ $ ) $ **'11ndhhz"#G&{EGOE% IU| c] 6hhzB/G E'E/$:Q$>?IcNIcN   r c|jjddt|}t|}t t }|t jdt|z }t }g}g}|D]}t d} d} ||jD]\\} } } | | | dz| z<| | | zz } |jd}|t jdt||zz }|t jdtj| z }t| }||z }|jt||jt!||t|t#j$t|d z}|j'|t)j*|j-}|j/d j'|t t0}||z }|D]}||z } |j/d j'|t3||}|j/d j'|t|t|t|t|d S)aEWrite the model artifacts as one digest-locked set. ``models.bin`` is written at *models_path* (whatever its basename); ``rowmax.bin`` and ``idf.bin`` are written beside it under their fixed names. ``rowmax.bin`` embeds the SHA-256 of the exact ``models.bin`` bytes written here and is deliberately written **last**: it is the commit marker for the whole set, so an interrupted run leaves a digest mismatch that :func:`parse_rowmax_bin` rejects at load time instead of a silently stale sibling (``idf.bin`` is validated by size only). :param models: Mapping of model key to sparse ``{(b1, b2): weight}``. :param models_path: Destination path for models.bin. :returns: Sizes in bytes, keyed by canonical artifact name (``models.bin``, ``rowmax.bin``, ``idf.bin``, ``langnorm.bin``). T)parentsexist_okz!Ir rr-r,!d idf.bin rowmax.bin langnorm.bin)z models.binryrxrz)parentmkdirsortedrrrr.r5packrr_encoderNrOrr3rYrRrcompress write_byteshashlibsha256digest with_namerAlangnorm_blob_for)rrZ sorted_namesroheaderr` rowmax_rows langnormsr:rsq_sumrkrlrd name_bytesfrozen models_blobr> rowmax_blobrow langnorm_blobs r'write_model_artifactsr,sH&TD9&>L6"I | $F fkk$L 122F [F!KIA%  &t 2 2 4 & HRf$*E27b. ! fvo %F &[[) &++dC O4zAA&++dDIIf$566u&,V45,VY?@A"-$--f q"AAKK(NN;/668M)$00;L)K= Ks ,'33K@%i?M.)55mD+&+&y>M*  r rctt}||z }|D]}|tjd|z }t |S)zblobr;s r'rrpsD ^ $DMD(  D$''( ;r )i)!__doc__rrNr5rpathlibrr.rArBrIrJStruct unpack_fromr/r2rintliststrdictr(tuplefloatr=rGrKrRrYrerrrrr r'rsD(    t$00&--%11CIJ JJ%)#YJ