💫 Industrial-strength Natural Language Processing (NLP) in Python
-
Updated
Aug 1, 2024 - Python
💫 Industrial-strength Natural Language Processing (NLP) in Python
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
Easy token price estimates for 400+ LLMs. TokenOps.
Secure Vault for Customer PII/PHI/PCI/KYC Records
Ravencoin Core integration/staging tree
Unsupervised text tokenizer focused on computational efficiency
👑 spaCy building blocks and visualizers for Streamlit apps
All the slides, accompanying code and exercises all stored in this repo. 🎈
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
🎤 vibrato: Viterbi-based accelerated tokenizer
Sudachi in Rust 🦀 and new generation of SudachiPy
Fast and customizable text tokenization library with BPE and SentencePiece support
CodeChain's official implementation in Rust.
Rule-based token, sentence segmentation for Russian language
TokenScript schema, specs and paper
Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.
To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."