mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-12 04:01:35 +00:00
docs: premium UI improvements — navbar links, hover effects, inline tips, accordion troubleshooting (#646)
- Move Discord, GitHub, PyPI, and Follow on X links from sidebar anchors to top-right navbar - Lock dark mode as default via appearance.strict and hide theme toggle - Add custom.css with hover highlighting for tables, code blocks, cards, callouts, and inline code - Move all Tips and Common Pitfalls sections inline next to their relevant content across all 25 reference docs - Polish context.md: remove duplicates, condense callouts, upgrade Cookbooks to CardGroup - Convert Troubleshooting and Performance Optimization sections in installation.md, cli-setup.md, explorer-setup.md, learning-more.md, and faq.md from plain headers to AccordionGroup - Change navigation-hint Tip callouts to Info in concepts.md, faq.md, glossary.md, and modules.md
This commit is contained in:
+12
-14
@@ -179,6 +179,10 @@ splitter = TextSplitter(
|
||||
| `relation_method` | `str` | `"ml"` | Relation extraction method for `relation_aware`: `"ml"` \| `"llm"` \| `"huggingface"` |
|
||||
| `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method: unrecognised names fall back to `cl100k_base` |
|
||||
|
||||
<Warning>
|
||||
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
|
||||
</Warning>
|
||||
|
||||
## Splitting Method Details
|
||||
|
||||
<Tabs>
|
||||
@@ -217,6 +221,10 @@ splitter = TextSplitter(
|
||||
- Produces variable-length chunks: some topics are short, others long
|
||||
- Falls back to sentence splitting if `sentence-transformers` is not installed
|
||||
- Slower than `recursive` due to embedding computation; cache embeddings for repeated splits
|
||||
|
||||
<Tip>
|
||||
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead.
|
||||
</Tip>
|
||||
</Tab>
|
||||
<Tab title="Entity-Aware">
|
||||
Runs NER internally, then adjusts chunk boundaries so no entity mention is split across two chunks:
|
||||
@@ -346,6 +354,10 @@ The `token` method accepts a `tokenizer=` kwarg that is passed to `tiktoken.enco
|
||||
|
||||
If `tiktoken` is not installed, the `token` method falls back to splitting by whitespace-separated words.
|
||||
|
||||
<Warning>
|
||||
**Wrong tokenizer.** The `token` method passes the `tokenizer=` value to `tiktoken.encoding_for_model()`. If the model name is not recognised by tiktoken it silently falls back to `cl100k_base`. Pass a valid tiktoken model name (e.g. `"gpt-4"`, `"gpt-3.5-turbo"`) to get deterministic behaviour.
|
||||
</Warning>
|
||||
|
||||
## Pipeline Integration
|
||||
|
||||
`TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern: parse a file, split the text, then extract entities from each chunk:
|
||||
@@ -373,20 +385,6 @@ for chunk in chunks:
|
||||
|
||||
For the full pipeline orchestration API, see the [Pipeline reference](pipeline).
|
||||
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**Wrong tokenizer.** The `token` method passes the `tokenizer=` value to `tiktoken.encoding_for_model()`. If the model name is not recognised by tiktoken it silently falls back to `cl100k_base`. Pass a valid tiktoken model name (e.g. `"gpt-4"`, `"gpt-3.5-turbo"`) to get deterministic behaviour.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead.
|
||||
</Tip>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Parse" icon="file-lines" href="parse">
|
||||
Parse documents before chunking: produces sections and metadata.
|
||||
|
||||
Reference in New Issue
Block a user