| Server IP : 172.24.0.40 / Your IP : 216.73.216.10 Web Server : Apache System : Linux dbweb26.ust.edu.ph 4.18.0-513.5.1.el8_9.x86_64 #1 SMP Fri Sep 29 05:21:10 EDT 2023 x86_64 User : apache ( 48) PHP Version : 8.2.18 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : ON Directory : /usr/lib/python3.6/site-packages/chardet/__pycache__/ |
Upload File : |
3
�]9Y� � @ s0 d dl Z d dlZddlmZ G dd� de�ZdS )� N� )� ProbingStatec @ sn e Zd ZdZddd�Zdd� Zedd� �Zd d
� Zed d � �Z d
d� Z
e dd� �Z e dd� �Z
e dd� �ZdS )�
CharSetProbergffffff�?Nc C s d | _ || _tjt�| _d S )N)�_state� lang_filter�logging� getLogger�__name__�logger)�selfr � r �#/usr/lib/python3.6/charsetprober.py�__init__' s zCharSetProber.__init__c C s t j| _d S )N)r Z DETECTINGr )r r r r
�reset, s zCharSetProber.resetc C s d S )Nr )r r r r
� charset_name/ s zCharSetProber.charset_namec C s d S )Nr )r �bufr r r
�feed3 s zCharSetProber.feedc C s | j S )N)r )r r r r
�state6 s zCharSetProber.statec C s dS )Ng r )r r r r
�get_confidence: s z CharSetProber.get_confidencec C s t jdd| �} | S )Ns ([ -])+� )�re�sub)r r r r
�filter_high_byte_only= s z#CharSetProber.filter_high_byte_onlyc C sb t � }tjd| �}xJ|D ]B}|j|dd� � |dd� }|j� rP|dk rPd}|j|� qW |S )u9
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [-ÿ]
marker: everything else [^a-zA-Z-ÿ]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
s% [a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?Nr � �r ���r )� bytearrayr �findall�extend�isalpha)r �filteredZwordsZwordZ last_charr r r
�filter_international_wordsB s
z(CharSetProber.filter_international_wordsc C s� t � }d}d}x�tt| ��D ]r}| ||d � }|dkr>d}n |dkrJd}|dk r |j� r ||kr�| r�|j| ||� � |jd� |d }q W |s�|j| |d � � |S )
a�
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
Also retains English alphabet and high byte characters immediately
before occurrences of >.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
Fr r � >� <Tr r N)r �range�lenr r )r r Zin_tag�prevZcurrZbuf_charr r r
�filter_with_english_lettersg s"
z)CharSetProber.filter_with_english_letters)N)r �
__module__� __qualname__ZSHORTCUT_THRESHOLDr r �propertyr r r r � staticmethodr r r& r r r r
r # s
%r )r r Zenumsr �objectr r r r r
�<module> s