# Nasty User Guide
A comprehensive guide to using the Nasty NLP library for natural language processing in Elixir.
## Table of Contents
1. [Introduction](#introduction)
2. [Installation](#installation)
3. [Quick Start](#quick-start)
4. [Core Concepts](#core-concepts)
5. [Basic Text Processing](#basic-text-processing)
6. [Phrase and Sentence Parsing](#phrase-and-sentence-parsing)
7. [Semantic Analysis](#semantic-analysis)
8. [Advanced NLP Operations](#advanced-nlp-operations)
9. [Code Interoperability](#code-interoperability)
10. [AST Manipulation](#ast-manipulation)
11. [Visualization and Debugging](#visualization-and-debugging)
12. [Statistical Models](#statistical-models)
13. [Performance Tips](#performance-tips)
14. [Troubleshooting](#troubleshooting)
## Introduction
Nasty (Natural Abstract Syntax Treey) is a comprehensive NLP library that treats natural language with the same rigor as programming languages. It provides a complete grammatical Abstract Syntax Tree (AST) for English, enabling sophisticated text analysis and manipulation.
### Key Features
- **Complete NLP Pipeline**: From tokenization to summarization
- **Grammar-First Design**: Linguistically rigorous AST structure
- **Statistical Models**: HMM POS tagger with 95% accuracy
- **Bidirectional Code Conversion**: Natural language ↔ Elixir code
- **AST Utilities**: Traversal, querying, validation, and transformation
- **Visualization**: Export to DOT/Graphviz and JSON formats
## Installation
Add `nasty` to your dependencies in `mix.exs`:
```elixir
def deps do
[
{:nasty, "~> 0.1.0"}
]
end
```
Then run:
```bash
mix deps.get
```
## Quick Start
Here's a simple example to get started:
```elixir
alias Nasty.Language.English
# Parse a sentence
text = "The quick brown fox jumps over the lazy dog."
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, document} = English.parse(tagged)
# Extract information
alias Nasty.Utils.Query
# Count tokens
token_count = Query.count(document, :token)
# => 9
# Find all nouns
nouns = Query.find_by_pos(document, :noun)
# => [%Token{text: "fox", ...}, %Token{text: "dog", ...}]
# Render back to text
alias Nasty.Rendering.Text
{:ok, text} = Text.render(document)
# => "The quick brown fox jumps over the lazy dog."
```
## Core Concepts
### AST Structure
Nasty represents text as a hierarchical tree structure:
```
Document
└── Paragraph
└── Sentence
└── Clause
├── Subject (NounPhrase)
│ ├── Determiner (Token)
│ ├── Modifiers (Tokens)
│ └── Head (Token)
└── Predicate (VerbPhrase)
├── Auxiliaries (Tokens)
├── Head (Token)
└── Complements (NounPhrases, etc.)
```
### Universal Dependencies
All POS tags and dependency relations follow the Universal Dependencies standard:
**POS Tags**: `noun`, `verb`, `adj`, `adv`, `det`, `adp`, `aux`, `cconj`, `sconj`, `pron`, `propn`, `num`, `punct`
**Dependencies**: `nsubj`, `obj`, `iobj`, `amod`, `advmod`, `det`, `case`, `acl`, `advcl`, `conj`, `cc`
### Language Markers
Every AST node carries a language identifier (`:en` for English), enabling future multilingual support.
## Basic Text Processing
### Tokenization
Split text into tokens (words and punctuation):
```elixir
alias Nasty.Language.English
text = "Hello, world! How are you?"
{:ok, tokens} = English.tokenize(text)
# Tokens include position information
Enum.each(tokens, fn token ->
IO.puts("#{token.text} at #{inspect(token.span)}")
end)
```
### POS Tagging
Assign grammatical categories to tokens:
```elixir
# Rule-based tagging (fast, ~85% accuracy)
{:ok, tagged} = English.tag_pos(tokens)
# Statistical tagging (higher accuracy, ~95%)
{:ok, tagged} = English.tag_pos(tokens, model: :hmm)
# Neural tagging (best accuracy, 97-98%)
{:ok, tagged} = English.tag_pos(tokens, model: :neural)
# Ensemble (combines all models)
{:ok, tagged} = English.tag_pos(tokens, model: :ensemble)
# Inspect tags
Enum.each(tagged, fn token ->
IO.puts("#{token.text}: #{token.pos_tag}")
end)
```
### Morphological Analysis
Extract lemmas and morphological features:
```elixir
alias Nasty.Language.English.Morphology
tagged
|> Enum.map(fn token ->
lemma = Morphology.lemmatize(token.text, token.pos_tag)
features = Morphology.extract_features(token.text, token.pos_tag)
{token.text, lemma, features}
end)
|> Enum.each(fn {text, lemma, features} ->
IO.puts("#{text} -> #{lemma} (#{inspect(features)})")
end)
```
## Phrase and Sentence Parsing
### Building the AST
Parse tokens into a complete AST:
```elixir
text = "The cat sat on the mat."
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, document} = English.parse(tagged)
# Access structure
paragraph = List.first(document.paragraphs)
sentence = List.first(paragraph.sentences)
IO.puts("Sentence type: #{sentence.function}, #{sentence.structure}")
```
### Phrase Structure
Extract and analyze phrases:
```elixir
alias Nasty.Utils.Query
# Find all noun phrases
noun_phrases = Query.find_all(document, :noun_phrase)
Enum.each(noun_phrases, fn np ->
det = if np.determiner, do: np.determiner.text, else: ""
mods = Enum.map(np.modifiers, & &1.text) |> Enum.join(" ")
head = np.head.text
IO.puts("NP: #{det} #{mods} #{head}")
end)
# Find verb phrases
verb_phrases = Query.find_all(document, :verb_phrase)
Enum.each(verb_phrases, fn vp ->
aux = Enum.map(vp.auxiliaries, & &1.text) |> Enum.join(" ")
verb = vp.head.text
IO.puts("VP: #{aux} #{verb}")
end)
```
### Sentence Structure Analysis
Analyze sentence complexity:
```elixir
document.paragraphs
|> Enum.flat_map(& &1.sentences)
|> Enum.each(fn sentence ->
IO.puts("Function: #{sentence.function}")
IO.puts("Structure: #{sentence.structure}")
IO.puts("Clauses: #{1 + length(sentence.additional_clauses)}")
IO.puts("")
end)
```
### Dependency Relations
Extract grammatical dependencies:
```elixir
alias Nasty.Language.English.DependencyExtractor
sentences = document.paragraphs |> Enum.flat_map(& &1.sentences)
Enum.each(sentences, fn sentence ->
deps = DependencyExtractor.extract(sentence)
Enum.each(deps, fn dep ->
IO.puts("#{dep.head.text} --#{dep.relation}--> #{dep.dependent.text}")
end)
end)
```
## Semantic Analysis
### Named Entity Recognition
Extract and classify named entities:
```elixir
alias Nasty.Language.English.EntityRecognizer
text = "John Smith works at Google in New York."
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
entities = EntityRecognizer.recognize(tagged)
Enum.each(entities, fn entity ->
IO.puts("#{entity.text}: #{entity.type} (confidence: #{entity.confidence})")
end)
# => John Smith: PERSON (confidence: 0.8)
# Google: ORG (confidence: 0.8)
# New York: GPE (confidence: 0.7)
```
### Semantic Role Labeling
Identify who did what to whom:
```elixir
{:ok, document} = English.parse(tagged, semantic_roles: true)
document.semantic_frames
|> Enum.each(fn frame ->
IO.puts("Predicate: #{frame.predicate}")
Enum.each(frame.roles, fn role ->
IO.puts(" #{role.type}: #{role.text}")
end)
end)
# => Predicate: works
# agent: John Smith
# location: at Google
```
### Coreference Resolution
Link mentions across sentences:
```elixir
text = """
John Smith is a software engineer. He works at Google.
The company is based in Mountain View.
"""
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, document} = English.parse(tagged, coreference: true)
document.coref_chains
|> Enum.each(fn chain ->
IO.puts("Representative: #{chain.representative.text}")
IO.puts("Mentions: #{Enum.map(chain.mentions, & &1.text) |> Enum.join(", ")}")
end)
# => Representative: John Smith
# Mentions: John Smith, He
```
## Advanced NLP Operations
### Text Summarization
Extract key sentences from documents:
```elixir
alias Nasty.Language.English
alias Nasty.Rendering.Text
long_text = """
[Your long document here...]
"""
{:ok, tokens} = English.tokenize(long_text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, document} = English.parse(tagged)
# Extractive summarization - returns list of Sentence structs
summary_sentences = English.summarize(document, ratio: 0.3)
IO.puts("30% summary (#{length(summary_sentences)} sentences):")
# Render summary sentences to text
Enum.each(summary_sentences, fn sentence ->
{:ok, text} = Text.render(sentence)
IO.puts(text)
end)
# Fixed sentence count
summary_sentences = English.summarize(document, max_sentences: 3)
# MMR for reduced redundancy
summary_sentences = English.summarize(document,
max_sentences: 3,
method: :mmr,
mmr_lambda: 0.5
)
```
### Question Answering
Answer questions from documents:
```elixir
text = """
John Smith is a software engineer at Google.
He graduated from Stanford University in 2010.
Google is headquartered in Mountain View, California.
"""
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, document} = English.parse(tagged)
# Ask questions
questions = [
"Who works at Google?",
"Where is Google located?",
"When did John Smith graduate?",
"What is John Smith's profession?"
]
Enum.each(questions, fn question ->
{:ok, answers} = English.answer_question(document, question)
IO.puts("Q: #{question}")
Enum.each(answers, fn answer ->
IO.puts("A: #{answer.text} (confidence: #{answer.confidence})")
end)
IO.puts("")
end)
```
### Text Classification
Train and apply classifiers:
```elixir
alias Nasty.Language.English
# Prepare training data
positive_reviews = [
"This product is amazing! Highly recommended.",
"Excellent quality and fast shipping.",
"Love it! Best purchase ever."
]
negative_reviews = [
"Terrible product. Waste of money.",
"Poor quality and slow delivery.",
"Very disappointed with this purchase."
]
# Parse documents
training_data =
Enum.map(positive_reviews, fn text ->
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, doc} = English.parse(tagged)
{doc, :positive}
end) ++
Enum.map(negative_reviews, fn text ->
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, doc} = English.parse(tagged)
{doc, :negative}
end)
# Train classifier
model = English.train_classifier(training_data,
features: [:bow, :lexical]
)
# Classify new text
test_text = "Great product, very satisfied!"
{:ok, tokens} = English.tokenize(test_text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, doc} = English.parse(tagged)
{:ok, predictions} = English.classify(doc, model)
IO.inspect(predictions)
```
### Information Extraction
Extract structured information:
```elixir
text = """
Apple Inc. acquired Beats Electronics for $3 billion in 2014.
The company is headquartered in Cupertino, California.
Tim Cook serves as CEO of Apple.
"""
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, document} = English.parse(tagged)
# Extract relations
{:ok, relations} = English.extract_relations(document)
Enum.each(relations, fn rel ->
IO.puts("#{rel.subject.text} --#{rel.type}--> #{rel.object.text}")
end)
# Extract events
{:ok, events} = English.extract_events(document)
Enum.each(events, fn event ->
IO.puts("Event: #{event.type}")
IO.puts("Trigger: #{event.trigger}")
IO.puts("Participants: #{inspect(event.participants)}")
end)
# Template-based extraction
alias Nasty.Language.English.TemplateExtractor
templates = [
TemplateExtractor.employment_template(),
TemplateExtractor.acquisition_template()
]
{:ok, results} = English.extract_templates(document, templates)
Enum.each(results, fn result ->
IO.puts("Template: #{result.template}")
IO.puts("Slots: #{inspect(result.slots)}")
end)
```
## Code Interoperability
### Natural Language to Code
Convert natural language commands to Elixir code:
```elixir
alias Nasty.Language.English
# Simple operations
{:ok, code} = English.to_code("Sort the list")
IO.puts(code)
# => "Enum.sort(list)"
{:ok, code} = English.to_code("Filter users where age is greater than 18")
IO.puts(code)
# => "Enum.filter(users, fn item -> item > 18 end)"
{:ok, code} = English.to_code("Map the numbers to double each one")
IO.puts(code)
# => "Enum.map(numbers, fn item -> item * 2 end)"
# Get the AST
{:ok, ast} = English.to_code_ast("Sort the numbers")
IO.inspect(ast)
# Recognize intent without generating code
{:ok, intent} = English.recognize_intent("Filter the list")
IO.inspect(intent)
```
### Code to Natural Language
Explain code in natural language:
```elixir
alias Nasty.Language.English
# Explain code strings
{:ok, explanation} = English.explain_code("Enum.sort(numbers)")
IO.puts(explanation)
# => "sort numbers"
{:ok, explanation} = English.explain_code("""
list
|> Enum.map(&(&1 * 2))
|> Enum.filter(&(&1 > 10))
|> Enum.sum()
""")
IO.puts(explanation)
# => "map list to each element times 2, then filter list where item is greater than 10, then sum list"
# Explain from AST
code_ast = quote do: x = a + b
{:ok, doc} = English.explain_code_to_document(code_ast)
{:ok, text} = Nasty.Rendering.Text.render(doc)
IO.puts(text)
```
## Translation
### AST-Based Translation
Translate documents between languages while preserving grammatical structure:
```elixir
alias Nasty.Language.{English, Spanish}
alias Nasty.Translation.Translator
# English to Spanish
text_en = "The quick cat runs in the garden."
{:ok, tokens_en} = English.tokenize(text_en)
{:ok, tagged_en} = English.tag_pos(tokens_en)
{:ok, doc_en} = English.parse(tagged_en)
# Translate document
{:ok, doc_es} = Translator.translate_document(doc_en, :es)
# Render Spanish text
alias Nasty.Rendering.Text
{:ok, text_es} = Text.render(doc_es)
IO.puts(text_es)
# => "El gato rápido corre en el jardín."
# Or translate text directly
{:ok, text_es} = Translator.translate("The quick cat runs.", :en, :es)
IO.puts(text_es)
# => "El gato rápido corre."
# Spanish to English
text_es = "La casa grande está en la ciudad."
{:ok, tokens_es} = Spanish.tokenize(text_es)
{:ok, tagged_es} = Spanish.tag_pos(tokens_es)
{:ok, doc_es} = Spanish.parse(tagged_es)
{:ok, doc_en} = Translator.translate_document(doc_es, :en)
{:ok, text_en} = Text.render(doc_en)
IO.puts(text_en)
# => "The big house is in the city."
```
### How Translation Works
The translation system operates on AST structures, not raw text:
1. **Parse source text** to AST
2. **Transform AST nodes** to target language structure
3. **Translate tokens** using lemma-to-lemma mapping with POS tags
4. **Apply morphological agreement** (gender, number, person)
5. **Apply word order rules** (language-specific)
6. **Render** target AST to text
### Morphological Agreement
The system automatically handles agreement:
```elixir
alias Nasty.Translation.Translator
alias Nasty.Rendering.Text
# English: "the cats"
# Spanish: "los gatos" (masculine plural determiner + noun)
{:ok, doc_en} = Nasty.parse("The cats.", language: :en)
{:ok, doc_es} = Translator.translate_document(doc_en, :es)
{:ok, text_es} = Text.render(doc_es)
# => "Los gatos."
# English: "the big houses"
# Spanish: "las casas grandes" (feminine plural, adjective after noun)
{:ok, doc_en} = Nasty.parse("The big houses.", language: :en)
{:ok, doc_es} = Translator.translate_document(doc_en, :es)
{:ok, text_es} = Text.render(doc_es)
# => "Las casas grandes."
```
### Word Order Transformations
Language-specific word order is automatically applied:
```elixir
alias Nasty.Translation.Translator
alias Nasty.Rendering.Text
# English: Adjective before noun
# Spanish: Most adjectives after noun
{:ok, doc_en} = Nasty.parse("The red car.", language: :en)
{:ok, doc_es} = Translator.translate_document(doc_en, :es)
{:ok, text_es} = Text.render(doc_es)
# => "El carro rojo." (car red)
# Some adjectives stay before noun
{:ok, doc_en} = Nasty.parse("The good book.", language: :en)
{:ok, doc_es} = Translator.translate_document(doc_en, :es)
{:ok, text_es} = Text.render(doc_es)
# => "El buen libro." (good stays before)
```
### Roundtrip Translation
Translations preserve grammatical structure for roundtrips:
```elixir
alias Nasty.Translation.Translator
alias Nasty.Rendering.Text
original = "The cat runs quickly."
# English -> Spanish -> English
{:ok, doc_en} = Nasty.parse(original, language: :en)
{:ok, doc_es} = Translator.translate_document(doc_en, :es)
{:ok, doc_en2} = Translator.translate_document(doc_es, :en)
{:ok, result} = Text.render(doc_en2)
IO.puts(original)
IO.puts(result)
# Original: "The cat runs quickly."
# Result: "The cat runs quickly." (or close equivalent)
```
### Supported Language Pairs
Currently supported:
- English ↔ Spanish
- English ↔ Catalan
- Spanish ↔ Catalan (via English)
### Custom Lexicons
Extend lexicons with domain-specific vocabulary:
```elixir
# Lexicons are in priv/translation/lexicons/
# Format: en_es.exs, es_en.exs, etc.
# Add entries in priv/translation/lexicons/en_es.exs:
%{
noun: %{
"widget" => "dispositivo",
"gadget" => "aparato"
},
verb: %{
"deploy" => "desplegar",
"compile" => "compilar"
}
}
```
### Translation Limitations
**Current limitations:**
- Idiomatic expressions may not translate well
- Complex verb tenses may need manual review
- Cultural context not preserved
- Ambiguous words use first lexicon entry
**Best practices:**
- Translate sentence by sentence for best results
- Review translations for idiomatic expressions
- Extend lexicons for domain-specific terms
- Use for technical/formal text rather than creative writing
## AST Manipulation
### Traversal
Walk the AST tree:
```elixir
alias Nasty.Utils.Traversal
# Count all tokens
token_count = Traversal.reduce(document, 0, fn
%Nasty.AST.Token{}, acc -> acc + 1
_, acc -> acc
end)
# Collect all verbs
verbs = Traversal.collect(document, fn
%Nasty.AST.Token{pos_tag: :verb} -> true
_ -> false
end)
# Find first question
question = Traversal.find(document, fn
%Nasty.AST.Sentence{function: :interrogative} -> true
_ -> false
end)
# Transform tree (lowercase all text)
lowercased = Traversal.map(document, fn
%Nasty.AST.Token{} = token ->
%{token | text: String.downcase(token.text)}
node ->
node
end)
# Breadth-first traversal
nodes = Traversal.walk_breadth(document, [], fn node, acc ->
{:cont, [node | acc]}
end)
```
### Queries
High-level querying API:
```elixir
alias Nasty.Utils.Query
# Find by type
noun_phrases = Query.find_all(document, :noun_phrase)
sentences = Query.find_all(document, :sentence)
# Find by POS tag
nouns = Query.find_by_pos(document, :noun)
verbs = Query.find_by_pos(document, :verb)
# Find by text pattern
cats = Query.find_by_text(document, "cat")
words_starting_with_s = Query.find_by_text(document, ~r/^s/i)
# Find by lemma
runs = Query.find_by_lemma(document, "run") # Matches "run", "runs", "running"
# Extract entities
all_entities = Query.extract_entities(document)
people = Query.extract_entities(document, type: :PERSON)
organizations = Query.extract_entities(document, type: :ORG)
# Structural queries
subject = Query.find_subject(sentence)
verb = Query.find_main_verb(sentence)
objects = Query.find_objects(sentence)
# Count nodes
token_count = Query.count(document, :token)
sentence_count = Query.count(document, :sentence)
# Content vs function words
content_words = Query.content_words(document)
function_words = Query.function_words(document)
# Custom predicates
long_words = Query.filter(document, fn
%Nasty.AST.Token{text: text} -> String.length(text) > 7
_ -> false
end)
```
### Transformations
Modify AST structures:
```elixir
alias Nasty.Utils.Transform
# Case normalization
lowercased = Transform.normalize_case(document, :lower)
uppercased = Transform.normalize_case(document, :upper)
titled = Transform.normalize_case(document, :title)
# Remove punctuation
no_punct = Transform.remove_punctuation(document)
# Remove stop words
no_stops = Transform.remove_stop_words(document)
# Custom stop words
custom_stops = ["the", "a", "an"]
filtered = Transform.remove_stop_words(document, custom_stops)
# Lemmatize all tokens
lemmatized = Transform.lemmatize(document)
# Replace tokens
masked = Transform.replace_tokens(
document,
fn token -> token.pos_tag == :propn end,
fn token -> %{token | text: "[MASK]"} end
)
# Transformation pipelines
processed = Transform.pipeline(document, [
&Transform.normalize_case(&1, :lower),
&Transform.remove_punctuation/1,
&Transform.remove_stop_words/1,
&Transform.lemmatize/1
])
# Round-trip testing
{:ok, transformed} = Transform.round_trip_test(document, fn doc ->
Transform.normalize_case(doc, :lower)
end)
```
### Validation
Ensure AST integrity:
```elixir
alias Nasty.Utils.Validator
# Validate structure
case Validator.validate(document) do
{:ok, doc} -> IO.puts("Valid!")
{:error, reason} -> IO.puts("Invalid: #{reason}")
end
# Check validity (boolean)
if Validator.valid?(document) do
IO.puts("Document is valid")
end
# Validate spans
case Validator.validate_spans(document) do
:ok -> IO.puts("Spans are consistent")
{:error, reason} -> IO.puts("Span error: #{reason}")
end
# Validate language consistency
case Validator.validate_language(document) do
:ok -> IO.puts("Language is consistent")
{:error, reason} -> IO.puts("Language error: #{reason}")
end
# Validate and raise
Validator.validate!(document) # Raises on error
```
## Visualization and Debugging
### Pretty Printing
Debug AST structures:
```elixir
alias Nasty.Rendering.PrettyPrint
# Indented output
IO.puts(PrettyPrint.print(document))
# With colors
IO.puts(PrettyPrint.print(document, color: true))
# Limit depth
IO.puts(PrettyPrint.print(document, max_depth: 3))
# Show spans
IO.puts(PrettyPrint.print(document, show_spans: true))
# Tree-style output
IO.puts(PrettyPrint.tree(document))
# Statistics
IO.puts(PrettyPrint.stats(document))
```
### Graphviz Visualization
Export to DOT format for visual rendering:
```elixir
alias Nasty.Rendering.Visualization
# Parse tree
dot = Visualization.to_dot(document, type: :parse_tree)
File.write("parse_tree.dot", dot)
# Then: dot -Tpng parse_tree.dot -o parse_tree.png
# Dependency graph
deps_dot = Visualization.to_dot(sentence,
type: :dependencies,
rankdir: "LR"
)
File.write("dependencies.dot", deps_dot)
# Entity graph
entity_dot = Visualization.to_dot(document, type: :entities)
File.write("entities.dot", entity_dot)
# Custom options
dot = Visualization.to_dot(document,
type: :parse_tree,
rankdir: "TB",
show_pos_tags: true,
show_spans: false
)
```
### JSON Export
Export for web visualization:
```elixir
alias Nasty.Rendering.Visualization
# Export to JSON (for d3.js, etc.)
json = Visualization.to_json(document)
File.write("document.json", json)
# Can be loaded in JavaScript:
# fetch('document.json')
# .then(r => r.json())
# .then(data => visualize(data))
```
### Text Rendering
Convert AST back to text:
```elixir
alias Nasty.Rendering.Text
# Basic rendering
{:ok, text} = Text.render(document)
# Or use language-specific rendering
alias Nasty.Language.English
{:ok, text} = English.render(document)
# For specific languages
alias Nasty.Language.{Spanish, Catalan}
{:ok, text_es} = Spanish.render(document)
{:ok, text_ca} = Catalan.render(document)
```
## Statistical & Neural Models
### Using Pretrained Models
Load and use statistical and neural models:
```elixir
alias Nasty.Language.English
# Automatic loading (looks in priv/models/)
{:ok, tokens} = English.tokenize(text)
# HMM statistical model (~95% accuracy)
{:ok, tagged} = English.tag_pos(tokens, model: :hmm)
# Neural model (97-98% accuracy)
{:ok, tagged} = English.tag_pos(tokens, model: :neural)
# Ensemble mode (combines neural + HMM + rule-based)
{:ok, tagged} = English.tag_pos(tokens, model: :ensemble)
# PCFG statistical parsing
{:ok, document} = English.parse(tagged, model: :pcfg)
# CRF-based named entity recognition
alias Nasty.Language.English.EntityRecognizer
entities = EntityRecognizer.recognize(tagged, model: :crf)
```
### Training Custom Models
Train on your own data:
```bash
# Download Universal Dependencies data
wget https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-4611/ud-treebanks-v2.10.tgz
# Extract
tar -xzf ud-treebanks-v2.10.tgz
# Train HMM POS tagger (fast, 95% accuracy)
mix nasty.train.pos \
--corpus ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-train.conllu \
--test ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-test.conllu \
--output priv/models/en/my_hmm_model.model
# Train neural POS tagger (slower, 97-98% accuracy)
mix nasty.train.neural_pos \
--corpus ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-train.conllu \
--output priv/models/en/my_neural_model.axon \
--epochs 10 \
--batch-size 32
# Train PCFG parser
mix nasty.train.pcfg \
--corpus ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-train.conllu \
--test ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-test.conllu \
--output priv/models/en/my_pcfg.model \
--smoothing 0.001
# Train CRF for named entity recognition
mix nasty.train.crf \
--corpus data/ner_train.conllu \
--test data/ner_test.conllu \
--output priv/models/en/my_crf_ner.model \
--task ner \
--iterations 100
# Evaluate models
mix nasty.eval.pos \
--model priv/models/en/my_hmm_model.model \
--test ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-test.conllu
mix nasty.eval \
--model priv/models/en/my_pcfg.model \
--test ud-treebanks-v2.10/UD_English-EWT/en_ewt-ud-test.conllu \
--type pcfg
mix nasty.eval \
--model priv/models/en/my_crf_ner.model \
--test data/ner_test.conllu \
--type crf \
--task ner
```
For detailed training instructions:
- Neural models: [TRAINING_NEURAL.md](TRAINING_NEURAL.md)
- PCFG and CRF models: [STATISTICAL_MODELS.md](STATISTICAL_MODELS.md)
### Model Management
```bash
# List models
mix nasty.models list
# Inspect model
mix nasty.models inspect priv/models/en/pos_hmm_v1.model
# Compare models
mix nasty.models compare model1.model model2.model
```
## Performance Tips
### Batch Processing
Process multiple texts efficiently:
```elixir
alias Nasty.Language.English
texts = [
"First sentence.",
"Second sentence.",
"Third sentence."
]
# Process in parallel
results =
texts
|> Task.async_stream(fn text ->
with {:ok, tokens} <- English.tokenize(text),
{:ok, tagged} <- English.tag_pos(tokens),
{:ok, doc} <- English.parse(tagged) do
{:ok, doc}
end
end, max_concurrency: System.schedulers_online())
|> Enum.map(fn {:ok, result} -> result end)
```
### Selective Parsing
Skip expensive operations when not needed:
```elixir
# Basic parsing (no semantic analysis)
{:ok, doc} = English.parse(tokens)
# With semantic roles
{:ok, doc} = English.parse(tokens, semantic_roles: true)
# With coreference
{:ok, doc} = English.parse(tokens, coreference: true)
# Full pipeline
{:ok, doc} = English.parse(tokens,
semantic_roles: true,
coreference: true
)
```
### Caching
Cache parsed documents:
```elixir
defmodule MyApp.DocumentCache do
use Agent
def start_link(_) do
Agent.start_link(fn -> %{} end, name: __MODULE__)
end
def get_or_parse(text) do
Agent.get_and_update(__MODULE__, fn cache ->
case Map.fetch(cache, text) do
{:ok, doc} ->
{doc, cache}
:error ->
{:ok, tokens} = English.tokenize(text)
{:ok, tagged} = English.tag_pos(tokens)
{:ok, doc} = English.parse(tagged)
{doc, Map.put(cache, text, doc)}
end
end)
end
end
```
## Troubleshooting
### Common Issues
**Issue**: Parsing fails with long sentences
**Solution**: Break into smaller sentences or increase timeout
```elixir
# Split long text
sentences = String.split(text, ~r/[.!?]+/)
Enum.map(sentences, &English.parse/1)
```
**Issue**: Entity recognition misses entities
**Solution**: Train custom NER or add to dictionary
```elixir
# Add custom entity patterns
alias Nasty.Language.English.EntityRecognizer
# This is conceptual - check actual API
EntityRecognizer.add_pattern(:ORG, ~r/\b[A-Z][a-z]+ Inc\.\b/)
```
**Issue**: POS tagging accuracy is low
**Solution**: Use statistical model or ensemble
```elixir
# Use HMM model
{:ok, tagged} = English.tag_pos(tokens, model: :hmm)
# Or ensemble
{:ok, tagged} = English.tag_pos(tokens, model: :ensemble)
```
### Debugging Tips
1. **Visualize the AST**: Use pretty printing to understand structure
2. **Check spans**: Ensure position tracking is correct
3. **Validate**: Run validation to catch structural issues
4. **Incremental parsing**: Test each pipeline stage separately
```elixir
# Debug pipeline stage by stage
{:ok, tokens} = English.tokenize(text)
IO.inspect(tokens, label: "Tokens")
{:ok, tagged} = English.tag_pos(tokens)
IO.inspect(tagged, label: "Tagged")
{:ok, doc} = English.parse(tagged)
IO.puts(PrettyPrint.tree(doc))
```
### Getting Help
- Check the [API documentation](https://hexdocs.pm/nasty/)
- Review [PLAN.md](../PLAN.md) for architecture details
- See [examples/](../examples/) for working code
- Report issues on [GitHub](https://github.com/am-kantox/nasty/issues)
## Next Steps
- Explore the [examples/](../examples/) directory for more demos
- Read [STATISTICAL_MODELS.md](STATISTICAL_MODELS.md) for ML details
- Check [TRAINING_GUIDE.md](TRAINING_GUIDE.md) to train custom models
- See [INTEROP_GUIDE.md](INTEROP_GUIDE.md) for code conversion details
Happy parsing!
