mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
feat(repo): add initial open-source structure, development guide, roadmap, contribution guide, and pyproject.toml for SemantiCore modular toolkit
This commit is contained in:
KaifAhmad1
+287
@@ -0,0 +1,287 @@
|
||||
# 🤝 Contributing to SemantiCore
|
||||
|
||||
Thank you for your interest in contributing to SemantiCore! This document provides guidelines and information for contributors.
|
||||
|
||||
## 🎯 How to Contribute
|
||||
|
||||
### **Types of Contributions**
|
||||
|
||||
1. **🐛 Bug Reports**: Report bugs and issues
|
||||
2. **✨ Feature Requests**: Suggest new features and improvements
|
||||
3. **📝 Documentation**: Improve documentation and examples
|
||||
4. **💻 Code Contributions**: Submit code changes and improvements
|
||||
5. **🧪 Testing**: Add tests and improve test coverage
|
||||
6. **🌐 Community**: Help with community support and discussions
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### **Prerequisites**
|
||||
- Python 3.8 or higher
|
||||
- Git
|
||||
- Basic knowledge of Python and semantic web technologies
|
||||
|
||||
### **Development Setup**
|
||||
```bash
|
||||
# Fork and clone the repository
|
||||
git clone https://github.com/YOUR_USERNAME/semanticore.git
|
||||
cd semanticore
|
||||
|
||||
# Create virtual environment
|
||||
python -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
|
||||
# Install development dependencies
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Setup pre-commit hooks
|
||||
pre-commit install
|
||||
```
|
||||
|
||||
## 📝 Development Workflow
|
||||
|
||||
### **1. Create a Feature Branch**
|
||||
```bash
|
||||
git checkout -b feature/your-feature-name
|
||||
# or
|
||||
git checkout -b fix/your-bug-fix
|
||||
```
|
||||
|
||||
### **2. Make Your Changes**
|
||||
- Follow the coding standards (see below)
|
||||
- Add tests for new functionality
|
||||
- Update documentation as needed
|
||||
|
||||
### **3. Test Your Changes**
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=semanticore
|
||||
|
||||
# Run linting
|
||||
flake8 semanticore/
|
||||
mypy semanticore/
|
||||
|
||||
# Format code
|
||||
black semanticore/
|
||||
isort semanticore/
|
||||
```
|
||||
|
||||
### **4. Commit Your Changes**
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "feat: add new PDF processor functionality
|
||||
|
||||
- Add support for table extraction from PDFs
|
||||
- Implement metadata extraction
|
||||
- Add comprehensive tests
|
||||
- Update documentation"
|
||||
```
|
||||
|
||||
### **5. Push and Create Pull Request**
|
||||
```bash
|
||||
git push origin feature/your-feature-name
|
||||
```
|
||||
|
||||
## 📋 Coding Standards
|
||||
|
||||
### **Python Code Style**
|
||||
- Follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) style guidelines
|
||||
- Use [Black](https://black.readthedocs.io/) for code formatting
|
||||
- Use [isort](https://pycqa.github.io/isort/) for import sorting
|
||||
- Maximum line length: 88 characters (Black default)
|
||||
|
||||
### **Code Quality**
|
||||
- Use [flake8](https://flake8.pycqa.org/) for linting
|
||||
- Use [mypy](https://mypy.readthedocs.io/) for type checking
|
||||
- Maintain test coverage above 80%
|
||||
- Write docstrings for all public functions and classes
|
||||
|
||||
### **Commit Message Format**
|
||||
Use [Conventional Commits](https://www.conventionalcommits.org/) format:
|
||||
|
||||
```
|
||||
<type>[optional scope]: <description>
|
||||
|
||||
[optional body]
|
||||
|
||||
[optional footer(s)]
|
||||
```
|
||||
|
||||
**Types:**
|
||||
- `feat`: New feature
|
||||
- `fix`: Bug fix
|
||||
- `docs`: Documentation changes
|
||||
- `style`: Code style changes (formatting, etc.)
|
||||
- `refactor`: Code refactoring
|
||||
- `test`: Adding or updating tests
|
||||
- `chore`: Maintenance tasks
|
||||
|
||||
**Examples:**
|
||||
```
|
||||
feat(processors): add Excel file processor
|
||||
fix(core): resolve memory leak in knowledge graph builder
|
||||
docs(api): update API documentation for new features
|
||||
test(extraction): add tests for entity extraction
|
||||
```
|
||||
|
||||
## 🧪 Testing Guidelines
|
||||
|
||||
### **Test Structure**
|
||||
- Unit tests in `tests/unit/`
|
||||
- Integration tests in `tests/integration/`
|
||||
- Performance tests in `tests/performance/`
|
||||
- Test data in `tests/fixtures/`
|
||||
|
||||
### **Writing Tests**
|
||||
```python
|
||||
import pytest
|
||||
from semanticore.processors.document.pdf_processor import PDFProcessor
|
||||
|
||||
class TestPDFProcessor:
|
||||
def setup_method(self):
|
||||
self.processor = PDFProcessor({
|
||||
'extract_tables': True,
|
||||
'extract_images': False
|
||||
})
|
||||
|
||||
def test_can_process_pdf(self):
|
||||
"""Test that PDF processor can identify PDF files."""
|
||||
assert self.processor.can_process("document.pdf")
|
||||
assert not self.processor.can_process("document.txt")
|
||||
|
||||
def test_process_pdf(self, sample_pdf_path):
|
||||
"""Test PDF processing functionality."""
|
||||
result = self.processor.process(sample_pdf_path)
|
||||
assert result.content is not None
|
||||
assert len(result.metadata) > 0
|
||||
```
|
||||
|
||||
### **Test Requirements**
|
||||
- All new code must have corresponding tests
|
||||
- Maintain test coverage above 80%
|
||||
- Use descriptive test names
|
||||
- Include both positive and negative test cases
|
||||
- Mock external dependencies
|
||||
|
||||
## 📚 Documentation Guidelines
|
||||
|
||||
### **Code Documentation**
|
||||
- Use Google-style docstrings
|
||||
- Include type hints for all functions
|
||||
- Document all public APIs
|
||||
|
||||
```python
|
||||
def extract_entities(self, text: str) -> List[Entity]:
|
||||
"""Extract named entities from text.
|
||||
|
||||
Args:
|
||||
text: Input text to extract entities from.
|
||||
|
||||
Returns:
|
||||
List of extracted entities with confidence scores.
|
||||
|
||||
Raises:
|
||||
ValueError: If text is empty or None.
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
### **Documentation Updates**
|
||||
- Update README.md for new features
|
||||
- Add examples in `examples/` directory
|
||||
- Update API documentation
|
||||
- Create tutorials for complex features
|
||||
|
||||
## 🔍 Review Process
|
||||
|
||||
### **Pull Request Checklist**
|
||||
- [ ] Code follows style guidelines
|
||||
- [ ] Tests pass and coverage is maintained
|
||||
- [ ] Documentation is updated
|
||||
- [ ] Commit messages follow conventional format
|
||||
- [ ] No breaking changes (or clearly documented)
|
||||
|
||||
### **Review Guidelines**
|
||||
- Be respectful and constructive
|
||||
- Focus on code quality and functionality
|
||||
- Suggest improvements when possible
|
||||
- Test the changes locally if needed
|
||||
|
||||
## 🐛 Bug Reports
|
||||
|
||||
### **Bug Report Template**
|
||||
```markdown
|
||||
**Bug Description**
|
||||
Brief description of the bug.
|
||||
|
||||
**Steps to Reproduce**
|
||||
1. Step 1
|
||||
2. Step 2
|
||||
3. Step 3
|
||||
|
||||
**Expected Behavior**
|
||||
What you expected to happen.
|
||||
|
||||
**Actual Behavior**
|
||||
What actually happened.
|
||||
|
||||
**Environment**
|
||||
- OS: [e.g., Windows 10, macOS 12.0]
|
||||
- Python version: [e.g., 3.9.7]
|
||||
- SemantiCore version: [e.g., 0.1.0]
|
||||
|
||||
**Additional Information**
|
||||
Any other relevant information.
|
||||
```
|
||||
|
||||
## 💡 Feature Requests
|
||||
|
||||
### **Feature Request Template**
|
||||
```markdown
|
||||
**Feature Description**
|
||||
Brief description of the feature.
|
||||
|
||||
**Use Case**
|
||||
Why this feature would be useful.
|
||||
|
||||
**Proposed Implementation**
|
||||
How you think it could be implemented.
|
||||
|
||||
**Alternatives Considered**
|
||||
Other approaches you've considered.
|
||||
```
|
||||
|
||||
## 🏷️ Issue Labels
|
||||
|
||||
- `bug`: Something isn't working
|
||||
- `enhancement`: New feature or request
|
||||
- `documentation`: Improvements or additions to documentation
|
||||
- `good first issue`: Good for newcomers
|
||||
- `help wanted`: Extra attention is needed
|
||||
- `question`: Further information is requested
|
||||
- `wontfix`: This will not be worked on
|
||||
|
||||
## 🎉 Recognition
|
||||
|
||||
Contributors will be recognized in:
|
||||
- Repository contributors list
|
||||
- Release notes
|
||||
- Documentation acknowledgments
|
||||
- Community highlights
|
||||
|
||||
## 📞 Getting Help
|
||||
|
||||
- **Discussions**: [GitHub Discussions](https://github.com/semanticore/semanticore/discussions)
|
||||
- **Issues**: [GitHub Issues](https://github.com/semanticore/semanticore/issues)
|
||||
- **Discord**: [Community Discord](https://discord.gg/semanticore)
|
||||
- **Email**: team@semanticore.io
|
||||
|
||||
## 📄 License
|
||||
|
||||
By contributing to SemantiCore, you agree that your contributions will be licensed under the MIT License.
|
||||
|
||||
---
|
||||
|
||||
Thank you for contributing to SemantiCore! 🚀
|
||||
Binary file not shown.
Binary file not shown.
+289
@@ -0,0 +1,289 @@
|
||||
# 🗺️ SemantiCore Development Roadmap
|
||||
|
||||
## 🎯 Vision & Goals
|
||||
|
||||
**Mission**: Build the most comprehensive open-source semantic data transformation platform that bridges unstructured data and intelligent AI systems.
|
||||
|
||||
**Vision**: Enable anyone to transform any data format into intelligent, structured semantic knowledge graphs, embeddings, and ontologies for LLMs, Agents, RAG systems, and Knowledge Graphs.
|
||||
|
||||
---
|
||||
|
||||
## 📅 Release Timeline
|
||||
|
||||
### 🚀 Phase 1: Foundation (Months 1-3) - v0.1.0 to v0.3.0
|
||||
|
||||
#### **v0.1.0 - Core Framework** (Month 1)
|
||||
- [x] Project structure and architecture design
|
||||
- [x] Basic package configuration and dependencies
|
||||
- [x] Core engine framework
|
||||
- [x] Configuration management system
|
||||
- [x] Basic exception handling
|
||||
- [x] Development tools and CI/CD setup
|
||||
|
||||
#### **v0.2.0 - Basic Processors** (Month 2)
|
||||
- [ ] Document processors (PDF, DOCX, TXT)
|
||||
- [ ] Web processors (HTML, RSS)
|
||||
- [ ] Structured data processors (JSON, CSV)
|
||||
- [ ] Base processor architecture
|
||||
- [ ] Content extraction and metadata handling
|
||||
- [ ] Basic semantic extraction (entities, relationships)
|
||||
|
||||
#### **v0.3.0 - Semantic Foundation** (Month 3)
|
||||
- [ ] Triple extraction and generation
|
||||
- [ ] Basic ontology generation
|
||||
- [ ] Simple knowledge graph construction
|
||||
- [ ] Text embeddings generation
|
||||
- [ ] Basic vector storage integration
|
||||
- [ ] End-to-end processing pipeline
|
||||
|
||||
### 🔧 Phase 2: Core Features (Months 4-6) - v0.4.0 to v0.6.0
|
||||
|
||||
#### **v0.4.0 - Advanced Processing** (Month 4)
|
||||
- [ ] Advanced document formats (PPTX, XLSX, LaTeX)
|
||||
- [ ] Email and archive processing
|
||||
- [ ] Academic content processing (BibTeX, JATS)
|
||||
- [ ] Multi-modal content extraction
|
||||
- [ ] Cross-document linking
|
||||
- [ ] Temporal analysis
|
||||
|
||||
#### **v0.5.0 - Knowledge Graph Enhancement** (Month 5)
|
||||
- [ ] Advanced triple generation
|
||||
- [ ] Ontology alignment and mapping
|
||||
- [ ] Graph database integrations (Neo4j, Blazegraph)
|
||||
- [ ] SPARQL query generation
|
||||
- [ ] Graph analytics and reasoning
|
||||
- [ ] Knowledge graph validation
|
||||
|
||||
#### **v0.6.0 - Embeddings & Search** (Month 6)
|
||||
- [ ] Advanced embedding models
|
||||
- [ ] Semantic chunking
|
||||
- [ ] Vector database integrations (Pinecone, ChromaDB)
|
||||
- [ ] Semantic search capabilities
|
||||
- [ ] Multi-modal embeddings
|
||||
- [ ] Embedding optimization
|
||||
|
||||
### 🌐 Phase 3: Real-time & Streaming (Months 7-9) - v0.7.0 to v0.9.0
|
||||
|
||||
#### **v0.7.0 - Live Processing** (Month 7)
|
||||
- [ ] RSS/Atom feed processing
|
||||
- [ ] Real-time web scraping
|
||||
- [ ] Stream processing integration
|
||||
- [ ] Kafka and RabbitMQ support
|
||||
- [ ] Live knowledge graph updates
|
||||
- [ ] Real-time semantic extraction
|
||||
|
||||
#### **v0.8.0 - API & Integration** (Month 8)
|
||||
- [ ] RESTful API development
|
||||
- [ ] GraphQL support
|
||||
- [ ] WebSocket real-time updates
|
||||
- [ ] Plugin architecture
|
||||
- [ ] Third-party integrations
|
||||
- [ ] API documentation and SDKs
|
||||
|
||||
#### **v0.9.0 - Domain Specialization** (Month 9)
|
||||
- [ ] Cybersecurity intelligence
|
||||
- [ ] Biomedical literature processing
|
||||
- [ ] Financial data analysis
|
||||
- [ ] Legal document processing
|
||||
- [ ] Academic research tools
|
||||
- [ ] Domain-specific ontologies
|
||||
|
||||
### 🚀 Phase 4: Enterprise & Scale (Months 10-12) - v1.0.0+
|
||||
|
||||
#### **v1.0.0 - Production Ready** (Month 10)
|
||||
- [ ] Enterprise deployment options
|
||||
- [ ] Kubernetes integration
|
||||
- [ ] Docker containerization
|
||||
- [ ] Monitoring and observability
|
||||
- [ ] Performance optimization
|
||||
- [ ] Security hardening
|
||||
|
||||
#### **v1.1.0 - Advanced Features** (Month 11)
|
||||
- [ ] Advanced reasoning capabilities
|
||||
- [ ] Machine learning pipeline integration
|
||||
- [ ] Automated quality assurance
|
||||
- [ ] Advanced analytics dashboard
|
||||
- [ ] Custom model training
|
||||
- [ ] Federated learning support
|
||||
|
||||
#### **v1.2.0 - Ecosystem** (Month 12)
|
||||
- [ ] Language model integrations (LangChain, Haystack)
|
||||
- [ ] RAG system optimizations
|
||||
- [ ] Agent orchestration
|
||||
- [ ] Marketplace for custom processors
|
||||
- [ ] Community plugins
|
||||
- [ ] Enterprise support tools
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Feature Priorities
|
||||
|
||||
### **High Priority (Must Have)**
|
||||
1. **Core Processing Engine**: Universal data ingestion and processing
|
||||
2. **Semantic Extraction**: Entity, relationship, and triple extraction
|
||||
3. **Knowledge Graph Construction**: Automated KG building from any data
|
||||
4. **Vector Embeddings**: Semantic embeddings for search and retrieval
|
||||
5. **Basic API**: RESTful API for core functionality
|
||||
|
||||
### **Medium Priority (Should Have)**
|
||||
1. **Real-time Processing**: Live data feed processing
|
||||
2. **Advanced Formats**: Support for complex document formats
|
||||
3. **Domain Specialization**: Industry-specific processors
|
||||
4. **Graph Analytics**: Advanced reasoning and analytics
|
||||
5. **Quality Assurance**: Automated validation and quality checks
|
||||
|
||||
### **Low Priority (Nice to Have)**
|
||||
1. **GUI Interface**: Web-based user interface
|
||||
2. **Advanced ML**: Custom model training capabilities
|
||||
3. **Federated Learning**: Distributed processing
|
||||
4. **Marketplace**: Plugin ecosystem
|
||||
5. **Enterprise Features**: Advanced security and compliance
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Technical Milestones
|
||||
|
||||
### **Architecture & Design**
|
||||
- [x] Modular architecture design
|
||||
- [x] Plugin system specification
|
||||
- [x] API design and documentation
|
||||
- [ ] Performance benchmarks
|
||||
- [ ] Scalability testing
|
||||
- [ ] Security audit
|
||||
|
||||
### **Core Components**
|
||||
- [ ] Data processing pipeline
|
||||
- [ ] Semantic extraction engine
|
||||
- [ ] Knowledge graph builder
|
||||
- [ ] Embedding generation system
|
||||
- [ ] Vector storage integration
|
||||
- [ ] Query and search interface
|
||||
|
||||
### **Quality & Testing**
|
||||
- [ ] Comprehensive test suite
|
||||
- [ ] Performance benchmarks
|
||||
- [ ] Security testing
|
||||
- [ ] Documentation coverage
|
||||
- [ ] Code quality metrics
|
||||
- [ ] Community testing
|
||||
|
||||
### **Deployment & Operations**
|
||||
- [ ] Docker containerization
|
||||
- [ ] Kubernetes manifests
|
||||
- [ ] CI/CD pipelines
|
||||
- [ ] Monitoring setup
|
||||
- [ ] Backup and recovery
|
||||
- [ ] Disaster recovery
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Community & Ecosystem
|
||||
|
||||
### **Documentation & Learning**
|
||||
- [ ] Comprehensive API documentation
|
||||
- [ ] Tutorial series and examples
|
||||
- [ ] Video tutorials and demos
|
||||
- [ ] Best practices guide
|
||||
- [ ] Performance optimization guide
|
||||
- [ ] Troubleshooting guide
|
||||
|
||||
### **Community Building**
|
||||
- [ ] Discord community server
|
||||
- [ ] GitHub discussions
|
||||
- [ ] Community meetups
|
||||
- [ ] Hackathons and workshops
|
||||
- [ ] Contributor recognition program
|
||||
- [ ] Mentorship program
|
||||
|
||||
### **Ecosystem Integration**
|
||||
- [ ] LangChain integration
|
||||
- [ ] Haystack integration
|
||||
- [ ] Streamlit templates
|
||||
- [ ] Jupyter notebook examples
|
||||
- [ ] VS Code extensions
|
||||
- [ ] Third-party integrations
|
||||
|
||||
---
|
||||
|
||||
## 📊 Success Metrics
|
||||
|
||||
### **Technical Metrics**
|
||||
- **Performance**: Process 1000+ documents/minute
|
||||
- **Accuracy**: 90%+ entity extraction accuracy
|
||||
- **Scalability**: Support 1M+ documents
|
||||
- **Reliability**: 99.9% uptime
|
||||
- **Coverage**: Support 50+ file formats
|
||||
|
||||
### **Community Metrics**
|
||||
- **GitHub Stars**: 1000+ stars
|
||||
- **Contributors**: 100+ contributors
|
||||
- **Downloads**: 10K+ monthly downloads
|
||||
- **Discussions**: Active community engagement
|
||||
- **Adoption**: Used in 100+ projects
|
||||
|
||||
### **Quality Metrics**
|
||||
- **Test Coverage**: 90%+ code coverage
|
||||
- **Documentation**: 100% API documented
|
||||
- **Performance**: <2s response time
|
||||
- **Security**: Zero critical vulnerabilities
|
||||
- **Accessibility**: WCAG 2.1 compliance
|
||||
|
||||
---
|
||||
|
||||
## 🚧 Current Development Status
|
||||
|
||||
### **In Progress**
|
||||
- [x] Repository structure setup
|
||||
- [x] Package configuration
|
||||
- [x] Development guidelines
|
||||
- [ ] Core engine implementation
|
||||
- [ ] Basic processor framework
|
||||
|
||||
### **Next Up**
|
||||
- [ ] PDF processor implementation
|
||||
- [ ] Basic semantic extraction
|
||||
- [ ] Triple generation
|
||||
- [ ] Knowledge graph builder
|
||||
- [ ] Vector embeddings
|
||||
|
||||
### **Blocked**
|
||||
- None currently
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing to the Roadmap
|
||||
|
||||
### **How to Contribute**
|
||||
1. **Review the roadmap** and identify areas of interest
|
||||
2. **Join discussions** on GitHub or Discord
|
||||
3. **Submit proposals** for new features
|
||||
4. **Implement features** following our guidelines
|
||||
5. **Share feedback** and suggestions
|
||||
|
||||
### **Priority Areas for Contributors**
|
||||
1. **Document Processors**: PDF, DOCX, PPTX, XLSX
|
||||
2. **Web Processors**: HTML, RSS, Web scraping
|
||||
3. **Semantic Extraction**: Entity and relationship extraction
|
||||
4. **Knowledge Graph**: Triple generation and storage
|
||||
5. **Examples & Documentation**: Tutorials and guides
|
||||
|
||||
### **Getting Started**
|
||||
- Check out our [Contributing Guide](CONTRIBUTING.md)
|
||||
- Join our [Discord Community](https://discord.gg/semanticore)
|
||||
- Review [open issues](https://github.com/semanticore/semanticore/issues)
|
||||
- Start with [good first issues](https://github.com/semanticore/semanticore/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
|
||||
|
||||
---
|
||||
|
||||
## 📞 Feedback & Suggestions
|
||||
|
||||
We welcome feedback and suggestions for the roadmap! Please:
|
||||
|
||||
- **Open an issue** for feature requests
|
||||
- **Join discussions** on GitHub
|
||||
- **Reach out** on Discord
|
||||
- **Email us** at roadmap@semanticore.io
|
||||
|
||||
---
|
||||
|
||||
*This roadmap is a living document and will be updated based on community feedback and development progress.*
|
||||
+391
@@ -0,0 +1,391 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "semanticore"
|
||||
version = "0.1.0"
|
||||
description = "Open Source Semantic Layer Toolkit - Transform any unstructured data into intelligent knowledge graphs"
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
authors = [
|
||||
{name = "SemantiCore Team", email = "team@semanticore.io"}
|
||||
]
|
||||
maintainers = [
|
||||
{name = "SemantiCore Team", email = "team@semanticore.io"}
|
||||
]
|
||||
keywords = [
|
||||
"semantic", "nlp", "knowledge-graph", "ai", "machine-learning",
|
||||
"data-processing", "embeddings", "ontology", "rdf", "sparql"
|
||||
]
|
||||
classifiers = [
|
||||
"Development Status :: 3 - Alpha",
|
||||
"Intended Audience :: Developers",
|
||||
"Intended Audience :: Science/Research",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: OS Independent",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.8",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||
"Topic :: Software Development :: Libraries :: Python Modules",
|
||||
"Topic :: Text Processing :: Linguistic",
|
||||
"Topic :: Database",
|
||||
"Topic :: Internet :: WWW/HTTP :: Dynamic Content",
|
||||
"Topic :: Scientific/Engineering :: Information Analysis",
|
||||
]
|
||||
|
||||
requires-python = ">=3.8"
|
||||
dependencies = [
|
||||
# Core dependencies
|
||||
"requests>=2.28.0",
|
||||
"beautifulsoup4>=4.11.0",
|
||||
"lxml>=4.9.0",
|
||||
"pandas>=1.5.0",
|
||||
"numpy>=1.21.0",
|
||||
"scikit-learn>=1.1.0",
|
||||
"pydantic>=1.10.0",
|
||||
"typing-extensions>=4.0.0",
|
||||
|
||||
# NLP and ML
|
||||
"transformers>=4.20.0",
|
||||
"torch>=1.12.0",
|
||||
"sentence-transformers>=2.2.0",
|
||||
"spacy>=3.4.0",
|
||||
"nltk>=3.8",
|
||||
|
||||
# Semantic Web
|
||||
"rdflib>=6.2.0",
|
||||
"owlready2>=0.44",
|
||||
"pyshacl>=0.20.0",
|
||||
|
||||
# Data processing
|
||||
"openpyxl>=3.0.10",
|
||||
"python-docx>=0.8.11",
|
||||
"python-pptx>=0.6.21",
|
||||
"PyPDF2>=3.0.0",
|
||||
"pdfplumber>=0.7.0",
|
||||
"feedparser>=6.0.0",
|
||||
"selenium>=4.0.0",
|
||||
"requests-html>=0.10.0",
|
||||
|
||||
# Vector databases
|
||||
"faiss-cpu>=1.7.0",
|
||||
"chromadb>=0.4.0",
|
||||
"pinecone-client>=2.2.0",
|
||||
|
||||
# Graph databases
|
||||
"neo4j>=5.0.0",
|
||||
"pymongo>=4.0.0",
|
||||
"redis>=4.0.0",
|
||||
|
||||
# Streaming and async
|
||||
"aiohttp>=3.8.0",
|
||||
"asyncio-mqtt>=0.11.0",
|
||||
"kafka-python>=2.0.0",
|
||||
|
||||
# Utilities
|
||||
"python-multipart>=0.0.5",
|
||||
"python-dateutil>=2.8.0",
|
||||
"pytz>=2022.1",
|
||||
"tqdm>=4.64.0",
|
||||
"click>=8.0.0",
|
||||
"rich>=12.0.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Document processing
|
||||
pdf = [
|
||||
"PyPDF2>=3.0.0",
|
||||
"pdfplumber>=0.7.0",
|
||||
"pdf2image>=1.16.0",
|
||||
"pymupdf>=1.22.0",
|
||||
]
|
||||
office = [
|
||||
"python-docx>=0.8.11",
|
||||
"openpyxl>=3.0.10",
|
||||
"python-pptx>=0.6.21",
|
||||
"xlrd>=2.0.1",
|
||||
]
|
||||
text = [
|
||||
"markdown>=3.4.0",
|
||||
"rst2html5>=1.0.0",
|
||||
"asciidoc>=10.0.0",
|
||||
]
|
||||
|
||||
# Web processing
|
||||
web = [
|
||||
"selenium>=4.0.0",
|
||||
"feedparser>=6.0.0",
|
||||
"requests-html>=0.10.0",
|
||||
"scrapy>=2.5.0",
|
||||
"newspaper3k>=0.2.8",
|
||||
]
|
||||
feeds = [
|
||||
"feedparser>=6.0.0",
|
||||
"requests-html>=0.10.0",
|
||||
"aiohttp>=3.8.0",
|
||||
]
|
||||
|
||||
# Structured data
|
||||
structured = [
|
||||
"openpyxl>=3.0.10",
|
||||
"xlrd>=2.0.1",
|
||||
"pyyaml>=6.0",
|
||||
"xmltodict>=0.13.0",
|
||||
"jsonschema>=4.0.0",
|
||||
]
|
||||
|
||||
# Email and archives
|
||||
email = [
|
||||
"email-validator>=1.3.0",
|
||||
"extract-msg>=0.41.0",
|
||||
"pypff>=20220101",
|
||||
]
|
||||
archives = [
|
||||
"patool>=1.12.0",
|
||||
"py7zr>=0.20.0",
|
||||
"rarfile>=4.0",
|
||||
]
|
||||
|
||||
# Academic and scientific
|
||||
academic = [
|
||||
"bibtexparser>=1.4.0",
|
||||
"scholarly>=1.7.0",
|
||||
"arxiv>=1.4.0",
|
||||
"crossref-commons>=0.0.7",
|
||||
]
|
||||
|
||||
# Database integrations
|
||||
database = [
|
||||
"neo4j>=5.0.0",
|
||||
"pymongo>=4.0.0",
|
||||
"redis>=4.0.0",
|
||||
"sqlalchemy>=1.4.0",
|
||||
"psycopg2-binary>=2.9.0",
|
||||
"pymysql>=1.0.0",
|
||||
]
|
||||
graphdb = [
|
||||
"neo4j>=5.0.0",
|
||||
"py2neo>=2021.0.0",
|
||||
"gremlinpython>=3.6.0",
|
||||
"amazon-neptune-python-utils>=1.0.0",
|
||||
]
|
||||
|
||||
# Vector stores
|
||||
vector = [
|
||||
"faiss-cpu>=1.7.0",
|
||||
"chromadb>=0.4.0",
|
||||
"pinecone-client>=2.2.0",
|
||||
"weaviate-client>=3.15.0",
|
||||
"qdrant-client>=1.1.0",
|
||||
"milvus>=2.2.0",
|
||||
]
|
||||
|
||||
# Machine learning
|
||||
ml = [
|
||||
"sentence-transformers>=2.2.0",
|
||||
"transformers>=4.20.0",
|
||||
"torch>=1.12.0",
|
||||
"tensorflow>=2.10.0",
|
||||
"scikit-learn>=1.1.0",
|
||||
"spacy>=3.4.0",
|
||||
"nltk>=3.8",
|
||||
"gensim>=4.2.0",
|
||||
]
|
||||
|
||||
# Streaming and real-time
|
||||
streaming = [
|
||||
"kafka-python>=2.0.0",
|
||||
"pika>=1.3.0",
|
||||
"aiohttp>=3.8.0",
|
||||
"websockets>=10.0",
|
||||
"asyncio-mqtt>=0.11.0",
|
||||
]
|
||||
|
||||
# Deployment and scaling
|
||||
deployment = [
|
||||
"kubernetes>=26.0.0",
|
||||
"docker>=6.0.0",
|
||||
"prometheus-client>=0.14.0",
|
||||
"grafana-api>=1.0.3",
|
||||
]
|
||||
|
||||
# Development dependencies
|
||||
dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-cov>=4.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"pytest-mock>=3.8.0",
|
||||
"black>=22.0.0",
|
||||
"isort>=5.10.0",
|
||||
"flake8>=5.0.0",
|
||||
"mypy>=0.991",
|
||||
"pre-commit>=2.20.0",
|
||||
"tox>=3.25.0",
|
||||
"coverage>=6.0.0",
|
||||
"bandit>=1.7.0",
|
||||
"safety>=2.0.0",
|
||||
]
|
||||
|
||||
# Documentation
|
||||
docs = [
|
||||
"sphinx>=5.0.0",
|
||||
"sphinx-rtd-theme>=1.0.0",
|
||||
"sphinx-autodoc-typehints>=1.19.0",
|
||||
"myst-parser>=0.18.0",
|
||||
"sphinx-copybutton>=0.5.0",
|
||||
]
|
||||
|
||||
# Testing
|
||||
test = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-cov>=4.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"pytest-mock>=3.8.0",
|
||||
"pytest-benchmark>=4.0.0",
|
||||
"factory-boy>=3.2.0",
|
||||
"faker>=18.0.0",
|
||||
]
|
||||
|
||||
# All optional dependencies
|
||||
all = [
|
||||
"semanticore[pdf,office,text,web,feeds,structured,email,archives,academic,database,graphdb,vector,ml,streaming,deployment]"
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://semanticore.io"
|
||||
Documentation = "https://semanticore.readthedocs.io"
|
||||
Repository = "https://github.com/semanticore/semanticore"
|
||||
"Bug Tracker" = "https://github.com/semanticore/semanticore/issues"
|
||||
Discussions = "https://github.com/semanticore/semanticore/discussions"
|
||||
Discord = "https://discord.gg/semanticore"
|
||||
Twitter = "https://twitter.com/semanticore"
|
||||
Blog = "https://blog.semanticore.io"
|
||||
|
||||
[project.scripts]
|
||||
semanticore = "semanticore.cli:main"
|
||||
|
||||
[project.gui-scripts]
|
||||
semanticore-gui = "semanticore.gui:main"
|
||||
|
||||
[tool.setuptools]
|
||||
packages = ["semanticore"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
semanticore = ["py.typed", "*.pyi"]
|
||||
|
||||
[tool.black]
|
||||
line-length = 88
|
||||
target-version = ['py38']
|
||||
include = '\.pyi?$'
|
||||
extend-exclude = '''
|
||||
/(
|
||||
# directories
|
||||
\.eggs
|
||||
| \.git
|
||||
| \.hg
|
||||
| \.mypy_cache
|
||||
| \.tox
|
||||
| \.venv
|
||||
| build
|
||||
| dist
|
||||
)/
|
||||
'''
|
||||
|
||||
[tool.isort]
|
||||
profile = "black"
|
||||
multi_line_output = 3
|
||||
line_length = 88
|
||||
known_first_party = ["semanticore"]
|
||||
known_third_party = ["pytest", "numpy", "pandas", "torch", "transformers"]
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.8"
|
||||
warn_return_any = true
|
||||
warn_unused_configs = true
|
||||
disallow_untyped_defs = true
|
||||
disallow_incomplete_defs = true
|
||||
check_untyped_defs = true
|
||||
disallow_untyped_decorators = true
|
||||
no_implicit_optional = true
|
||||
warn_redundant_casts = true
|
||||
warn_unused_ignores = true
|
||||
warn_no_return = true
|
||||
warn_unreachable = true
|
||||
strict_equality = true
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = [
|
||||
"torch.*",
|
||||
"transformers.*",
|
||||
"spacy.*",
|
||||
"nltk.*",
|
||||
"selenium.*",
|
||||
"scrapy.*",
|
||||
"kafka.*",
|
||||
"pika.*",
|
||||
"redis.*",
|
||||
"pymongo.*",
|
||||
"neo4j.*",
|
||||
"faiss.*",
|
||||
"chromadb.*",
|
||||
"pinecone.*",
|
||||
"weaviate.*",
|
||||
"qdrant.*",
|
||||
"milvus.*",
|
||||
"prometheus_client.*",
|
||||
"grafana_api.*",
|
||||
"kubernetes.*",
|
||||
"docker.*",
|
||||
]
|
||||
ignore_missing_imports = true
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
minversion = "6.0"
|
||||
addopts = "-ra -q --strict-markers --strict-config"
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py", "*_test.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
markers = [
|
||||
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
|
||||
"integration: marks tests as integration tests",
|
||||
"unit: marks tests as unit tests",
|
||||
"performance: marks tests as performance tests",
|
||||
]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["semanticore"]
|
||||
omit = [
|
||||
"*/tests/*",
|
||||
"*/test_*",
|
||||
"*/__pycache__/*",
|
||||
"*/venv/*",
|
||||
"*/env/*",
|
||||
"*/\.venv/*",
|
||||
]
|
||||
|
||||
[tool.coverage.report]
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"def __repr__",
|
||||
"if self.debug:",
|
||||
"if settings.DEBUG",
|
||||
"raise AssertionError",
|
||||
"raise NotImplementedError",
|
||||
"if 0:",
|
||||
"if __name__ == .__main__.:",
|
||||
"class .*\\bProtocol\\):",
|
||||
"@(abc\\.)?abstractmethod",
|
||||
]
|
||||
|
||||
[tool.bandit]
|
||||
exclude_dirs = ["tests", "docs"]
|
||||
skips = ["B101", "B601"]
|
||||
|
||||
[tool.safety]
|
||||
output = "json"
|
||||
Reference in New Issue
Block a user