o ~b@sddlmZddlmmZddlZddlZee dZ e dde de ddde dd e de dd diZ d e d e fd d ZdejfddZddZdejfddZdee dejd e fddZde dejfddZdS))ListNzflibl_config.json languages main_languagez([^Zvalid_charactersz])child_languagephraselgcCsr|r7ddddt||DD}g}d}|D]}|dkr$||7}q|r/|d|d}||q|SgS)zTokenize an utterance based on specified word-forming characters. Parameters: phrase: the string to be tokenized lg: the language whose word-forming characters are to be used in tokenization Return list of tokens cSsg|]}|r|qSr).0jrrM/Users/pat/Sites/forum-docling/projects/sunny-flibl/flexible-main/flexible.py ztokenize..cSsg|]}|qSr)stripr irrr r r . ) word_formingsplitappend)rrZtokensZcollected_tokensZ punct_charstokenrrr tokenizes "  relc Cs(td|jdt|jd|jdt|dS)zPrint the tag, attributes, text, and number of children of an ET.Element Parameters: el: Element to be printed zTag:z Attrs:z Text:z No. of Children:N)printtagstrattribtextlen)rrrr print_el_info)s(r cCsttdddd}t|dd}ddt||}|dd d |d d d |d d d |d d d |d d}tdddt||S)zGenerate FLEx guid based on offset defined in offset.txt Increments offset upon use. Return guid in format [0-f]{8}-([0-f]{4}-){3}[0-f]{12} z offset.txtr)modeN0 - w)intopenreadhexrwriter)Z global_offsetZ new_guid_numZ new_guid_strZnew_guidrrr generate_guid1s Lr2eaf_rootcCsdd|dDS)zGet time IDs and values from an EAF file Parameters: eaf_root: is the root element of an EAF object parsed through ElementTree Return a dictionary of time ID and value pairs cSsi|] }|jd|jdqS)Z TIME_SLOT_IDZ TIME_VALUErrrrr Hsztime_values..z .//TIME_SLOT)findall)r3rrr time_values@sr7 tokenized_utt phrase_elc Cstd}t|}|D]-}tjddtid}||rd}nd}tjd||dd}||_||||q ||d S) aPopulate a phrase element with a tokenized utterance *tokenized_utt* is a list with tokens from an utterance *phrase_el* is the element that will be the parent to the words added (below the words el in the phrase_el will be the items w/ translations and notes) *lg* is the language whose word-forming characters are to be used in tokenization Makes changes in place (returns nothing) wordswordguidr4punctZtxtitem)typelangN)ETElementrr2searchrr) r8r9rr:Zutterance_word_formingrr;r?Ztoken_elrrr add_word_elLs    rD constrainteafcCs@|d|D]}|d|jdD]}||qqdS)zRemove the tiers that use the "Included In" stereotype constraint Parameters: eaf: The root element of an EAF file Makes changes in place (returns nothing) z.//*[@CONSTRAINTS='{}']z.//*[@LINGUISTIC_TYPE_REF='{}']ZLINGUISTIC_TYPE_IDN)r6formatrremove)rErFZconstraint_tierZbad_tierrrr remove_constraintes   rI)typingrxml.etree.ElementTreeetree ElementTreerAjsonreloadr.configcompilerrrrBr r2r7rDrIrrrr s ""