Imported from chenzhu007/wework-mail-downloader (.trae/skills/autoresearch/SKILL.md). Install upstream with npx skills add chenzhu007/wework-mail-downloader --skill autoresearch. Copyright stays with the author.

AutoResearch - Automated Research and Information Gathering

This skill provides comprehensive guidance for conducting automated research, gathering information from multiple sources, and synthesizing data into actionable insights.

Core Principles

1. Multi-Source Research

• Query multiple search engines and databases
• Cross-reference information from different sources
• Validate findings across platforms
• Identify consensus and discrepancies

2. Intelligent Filtering

• Apply relevance scoring to results
• Filter out low-quality or outdated content
• Prioritize authoritative sources
• Remove duplicate information

3. Structured Data Extraction

• Extract key information systematically
• Organize data into structured formats
• Identify patterns and trends
• Create summaries and insights

4. Continuous Learning

• Learn from successful search strategies
• Adapt queries based on results
• Improve relevance over time
• Store useful sources for future reference

When to Use This Skill

Invoke this skill when:

• User needs comprehensive research on a topic
• Gathering information from multiple sources
• Conducting competitive analysis
• Researching best practices or solutions
• Finding documentation or resources
• Analyzing trends or market data
• Investigating issues or problems

Research Workflow

Phase 1: Query Planning

1. Understand the Goal: Clarify what information is needed
2. Identify Keywords: Extract key terms and concepts
3. Determine Scope: Define breadth and depth of research
4. Select Sources: Choose appropriate platforms and databases
5. Plan Strategy: Decide on search approach and filters

Phase 2: Information Gathering

1. Execute Searches: Run queries across multiple sources
2. Collect Results: Gather relevant pages and documents
3. Extract Data: Pull key information from sources
4. Track Sources: Record origin and credibility
5. Monitor Quality: Assess reliability of information

Phase 3: Analysis and Synthesis

1. Cross-Reference: Compare findings across sources
2. Identify Patterns: Recognize trends and commonalities
3. Resolve Conflicts: Address contradictory information
4. Synthesize Insights: Combine information into coherent picture
5. Generate Summary: Create concise overview of findings

Phase 4: Validation and Refinement

1. Verify Claims: Check accuracy of key statements
2. Update Outdated: Refresh time-sensitive information
3. Fill Gaps: Identify missing information
4. Refine Results: Improve clarity and completeness
5. Document Process: Record methodology and sources

Search Strategies

1. Broad-to-Narrow Approach

# Start with broad queries
"machine learning best practices"

# Narrow down based on results
"machine learning best practices python 2024"

# Further refine for specific needs
"machine learning best practices python scikit-learn production"

2. Multi-Modal Searching

• Web Search: General information and overviews
• Academic Databases: Research papers and studies
• Documentation Sites: Technical specifications and guides
• Forums and Communities: Practical experiences and solutions
• Code Repositories: Implementation examples and code

3. Temporal Filtering

# Add time constraints to queries
"react performance optimization 2024"
"python async await best practices recent"

# Use search engine filters
site:stackoverflow.com python async 2024

4. Source-Specific Queries

# Target specific platforms
site:docs.python.org asyncio
site:github.com react performance optimization
site:medium.com machine learning tutorial

# Use platform-specific syntax
"react hooks" tag:tutorial
"docker compose" label:best-practices

Information Sources

Web Search Engines

• Google: Comprehensive web search
• Bing: Alternative search perspective
• DuckDuckGo: Privacy-focused search
• Baidu: Chinese language content

Technical Resources

• Stack Overflow: Programming solutions
• GitHub: Code examples and repositories
• Documentation Sites: Official documentation
• API References: Technical specifications

Academic Sources

• Google Scholar: Research papers
• arXiv: Preprint research papers
• IEEE Xplore: Technical publications
• PubMed: Medical and life sciences

Professional Networks

• LinkedIn: Industry insights and trends
• ResearchGate: Academic collaboration
• Medium: Thought leadership and tutorials
• Dev.to: Developer community articles

Data Extraction Techniques

1. Web Scraping

# Extract structured data from web pages
from bs4 import BeautifulSoup
import requests

def extract_article_content(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    
    # Extract main content
    title = soup.find('h1').text
    content = soup.find('article').text
    
    return {
        'title': title,
        'content': content,
        'url': url
    }

2. API Integration

# Use APIs for structured data access
import requests

def search_github(query):
    url = f"https://api.github.com/search/repositories"
    params = {'q': query, 'sort': 'stars'}
    response = requests.get(url, params=params)
    
    return response.json()['items']

3. Content Parsing

• HTML Parsing: Extract text from web pages
• PDF Extraction: Read document content
• JSON Processing: Handle API responses
• XML Parsing: Process structured markup

4. Text Analysis

# Analyze and extract key information
import re
from collections import Counter

def extract_keywords(text):
    # Remove common words
    stop_words = {'the', 'and', 'is', 'in', 'at', 'of'}
    words = re.findall(r'\w+', text.lower())
    
    # Count word frequency
    word_count = Counter(word for word in words if word not in stop_words)
    
    return word_count.most_common(10)

Quality Assessment

Source Credibility

• Domain Authority: Check website reputation
• Author Expertise: Verify author credentials
• Publication Date: Ensure recency
• Citation Count: Measure impact (for academic sources)
• Peer Review: Check for validation process

Content Quality

• Accuracy: Verify factual correctness
• Completeness: Assess coverage of topic
• Clarity: Evaluate readability
• Relevance: Match to research goals
• Uniqueness: Identify original insights

Bias Detection

• Multiple Perspectives: Seek diverse viewpoints
• Source Diversity: Avoid single-source reliance
• Conflict Identification: Note contradictory information
• Context Analysis: Understand potential biases

Result Organization

1. Structured Formats

{
  "topic": "React Performance Optimization",
  "sources": [
    {
      "url": "https://example.com/article",
      "title": "Article Title",
      "credibility": "high",
      "date": "2024-01-15"
    }
  ],
  "key_findings": [
    "Finding 1",
    "Finding 2"
  ],
  "consensus": "Agreed upon points",
  "controversies": "Points of disagreement"
}

2. Categorization

• By Topic: Group related information
• By Source: Organize by origin
• By Credibility: Prioritize reliable sources
• By Date: Track information freshness

3. Summarization

# Generate concise summaries
def generate_summary(findings):
    # Extract key points
    key_points = [f['point'] for f in findings if f['importance'] > 0.7]
    
    # Create summary
    summary = {
        'overview': 'Brief description',
        'key_points': key_points[:5],
        'sources_count': len(findings),
        'confidence': calculate_confidence(findings)
    }
    
    return summary

Best Practices

DO:

• Start with clear research objectives
• Use multiple search strategies
• Verify information from diverse sources
• Document sources and methodology
• Update research regularly
• Respect copyright and terms of service
• Consider privacy and ethical implications
• Provide citations and references

DON'T:

• Rely on single sources
• Ignore publication dates
• Accept information without verification
• Overlook potential biases
• Violate website terms of service
• Spread misinformation
• Ignore conflicting information
• Skip quality assessment

Common Challenges and Solutions

1. Information Overload

Challenge: Too much information to process Solution:

• Implement relevance scoring
• Use advanced search filters
• Focus on high-quality sources
• Set time limits for research

2. Outdated Information

Challenge: Finding current and accurate data Solution:

• Use date-based filters
• Prioritize recent publications
• Check for update notices
• Verify with multiple recent sources

3. Conflicting Sources

Challenge: Different sources disagree Solution:

• Identify consensus areas
• Note specific disagreements
• Assess source credibility
• Seek additional authoritative sources
• Present multiple perspectives

4. Language Barriers

Challenge: Information in unfamiliar languages Solution:

• Use translation tools
• Search in multiple languages
• Focus on English for technical content
• Use language-specific search engines

5. Paywall Restrictions

Challenge: Access to premium content Solution:

• Look for open-access alternatives
• Check institutional access
• Use preprint servers
• Find summaries or secondary sources

Advanced Techniques

1. Semantic Search

# Use embeddings for semantic similarity
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def semantic_search(query, documents):
    query_embedding = model.encode(query)
    doc_embeddings = model.encode(documents)
    
    # Calculate similarities
    similarities = cosine_similarity(query_embedding, doc_embeddings)
    
    return sorted(zip(documents, similarities), key=lambda x: x[1], reverse=True)

2. Knowledge Graphs

• Entity Extraction: Identify key entities
• Relationship Mapping: Connect related concepts
• Graph Traversal: Explore connected information
• Pattern Recognition: Discover hidden relationships

3. Machine Learning Filtering

# Train relevance classifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

def train_relevance_classifier(training_data):
    vectorizer = TfidfVectorizer()
    X = vectorizer.fit_transform(training_data['text'])
    y = training_data['relevant']
    
    classifier = MultinomialNB()
    classifier.fit(X, y)
    
    return vectorizer, classifier

4. Automated Monitoring

# Set up continuous research
import schedule

def monitor_topic(topic):
    while True:
        # Search for new information
        new_info = search_topic(topic)
        
        # Process and alert if significant
        if is_significant(new_info):
            send_alert(new_info)
        
        schedule.run_pending()
        time.sleep(3600)  # Check hourly

Performance Optimization

1. Parallel Processing

from concurrent.futures import ThreadPoolExecutor

def parallel_search(queries):
    with ThreadPoolExecutor(max_workers=5) as executor:
        results = list(executor.map(execute_search, queries))
    
    return results

2. Caching Strategies

• Cache search results to avoid redundant queries
• Store frequently accessed information
• Implement TTL for cached data
• Use distributed caching for large-scale research

3. Rate Limiting

• Respect API rate limits
• Implement exponential backoff
• Queue requests when limits are reached
• Monitor usage patterns

Ethical Considerations

1. Privacy Protection

• Don't collect personal information without consent
• Anonymize data when possible
• Respect robots.txt and terms of service
• Follow data protection regulations

2. Intellectual Property

• Respect copyright laws
• Provide proper attribution
• Use content within fair use guidelines
• Link to original sources

3. Transparency

• Disclose automated research methods
• Provide source citations
• Acknowledge limitations
• Be honest about uncertainty

Tools and Technologies

Search APIs

• Google Custom Search API: Programmatic web search
• Bing Search API: Microsoft search services
• GitHub API: Code and repository search
• Stack Exchange API: Q&A platform data

Web Scraping Tools

• BeautifulSoup: HTML parsing
• Scrapy: Large-scale scraping
• Selenium: Dynamic content handling
• Playwright: Modern web automation

Data Processing

• Pandas: Data manipulation
• NumPy: Numerical computing
• NLTK/SpaCy: Natural language processing
• Transformers: Modern NLP models

Evaluation Metrics

Research Quality

• Source Diversity: Number of unique sources
• Credibility Score: Average source reliability
• Recency: Freshness of information
• Completeness: Coverage of research goals
• Accuracy: Verified correctness of findings

Efficiency Metrics

• Time to Results: Speed of information gathering
• Relevance Rate: Percentage of useful results
• Source Quality: Average credibility rating
• Reduction Ratio: Information compression achieved

User Satisfaction

• Helpfulness: Perceived value of research
• Completeness: Adequacy for user needs
• Clarity: Understandability of results
• Actionability: Usability of insights

Future Enhancements

Consider implementing:

• Real-time research monitoring
• Predictive research suggestions
• Collaborative research platforms
• AI-powered insight generation
• Automated fact-checking
• Cross-language research capabilities
• Visual research dashboards
• Integration with knowledge graphs