API Reference

PyKomodo provides several modules for document processing, chunking, and management.

Core Module

class pykomodo.core.PriorityRule[source]

Bases: object

__init__(pattern, score)[source]
class pykomodo.core.PyCConfig[source]

Bases: object

__init__()[source]
add_ignore_pattern(pattern)[source]
add_priority_rule(pattern, score)[source]
add_unignore_pattern(pattern)[source]
calculate_priority(path)[source]
count_tokens(text)[source]
is_binary_file(path)[source]
make_c_string(text)[source]
read_file_contents(path)[source]
should_ignore(path)[source]

Example usage:

from pykomodo.core import TextProcessor

processor = TextProcessor()
result = processor.process_text(my_document)

Multi-Directory Chunker

class pykomodo.multi_dirs_chunker.ChunkWriterInterface[source]

Bases: object

__init__(chunker)[source]
write_chunk(content_bytes, chunk_num)[source]
class pykomodo.multi_dirs_chunker.ParallelChunker[source]

Bases: object

__init__(equal_chunks=None, max_chunk_size=None, output_dir='chunks', user_ignore=None, user_unignore=None, binary_extensions=None, priority_rules=None, num_threads=4, dry_run=False, semantic_chunking=False, file_type=None, verbose=False, export_jsonl=False, export_path=None, export_embed_model=None)[source]
calculate_priority(path)[source]
close()[source]
is_absolute_pattern(pattern)[source]
is_binary_file(path)[source]
pdf_chunking(path, idx)[source]
process_directories(dirs)[source]
process_directory(directory)[source]
process_file(file_path, custom_chunk_size=None, force_process=False)[source]
should_ignore_file(path)[source]
DIR_IGNORE_NAMES = ['venv', '.venv', 'env', 'node_modules', '.git', '.svn', '.hg', '__pycache__', '.pytest_cache', '.tox', '.eggs', 'build', 'dist']
class pykomodo.multi_dirs_chunker.PriorityRule[source]

Bases: object

__init__(pattern, score)[source]

Example usage:

from pykomodo.multi_dirs_chunker import MultiDirChunker

chunker = MultiDirChunker()
chunks = chunker.chunk_directories(['path/to/dir1', 'path/to/dir2'], chunk_size=1000)

Enhanced Chunker

class pykomodo.enhanced_chunker.EnhancedParallelChunker[source]

Bases: ParallelChunker

__init__(equal_chunks=None, max_chunk_size=None, output_dir='chunks', user_ignore=None, user_unignore=None, binary_extensions=None, priority_rules=None, num_threads=4, extract_metadata=True, add_summaries=True, remove_redundancy=True, context_window=4096, min_relevance_score=0.3)[source]

Command Line Interface

pykomodo.command_line.main()[source]
pykomodo.command_line.run_server()[source]

Configuration

Example configuration:

from pykomodo.pykomodo_config import PyKomodoConfig

config = PyKomodoConfig()
config.load_config('config.yaml')