Module 2: The SCIC Framework Deep Dive

Overview

Now that you understand why context engineering transforms AI capabilities, it's time to master the framework that makes it systematic. SCIC (Select, Compress, Isolate, Compose) isn't just theory—it's a practical methodology you'll use in every context-engineered system.

What You'll Learn

How to SELECT relevant information intelligently
Techniques to COMPRESS without losing critical details
Ways to ISOLATE contexts for clarity
Methods to COMPOSE multiple contexts effectively

Prerequisites

Completed Module 1
Basic understanding of embeddings (we'll explain)
45 minutes for deep practice

SELECT: The Art of Intelligent Curation

The Selection Problem

Your AI needs context, but not ALL context. Imagine building a feature—what's actually relevant?

Project Files (1,847 files):
- Which 10 files matter for THIS task?
- Which functions within those files?
- Which documentation sections?
- Which past conversations?

Selection Strategies

1. Embedding-Based Selection

Think of embeddings as "meaning fingerprints":

# Simplified concept
query = "implement user authentication"

# Each document gets a "meaning score"
relevance_scores = {
    "auth_handler.py": 0.95,        # Highly relevant
    "user_model.py": 0.87,          # Very relevant
    "database_config.py": 0.72,     # Somewhat relevant
    "email_templates.py": 0.31,     # Less relevant
    "analytics.js": 0.12            # Barely relevant
}

# Select top N most relevant
selected_context = select_top_n(relevance_scores, n=5)

2. Recency Weighting

Recent context often matters more:

def calculate_relevance(document, query):
    semantic_score = get_embedding_similarity(document, query)
    time_decay = calculate_time_decay(document.last_modified)
    
    # Recent + relevant = higher priority
    return semantic_score * time_decay

# Example weights
# Today: 1.0x multiplier
# Yesterday: 0.9x multiplier
# Last week: 0.7x multiplier
# Last month: 0.4x multiplier

3. Dependency Tracking

Include related context automatically:

# If you select user_controller.py
# Also include:
- user_model.py (imported)
- auth_middleware.py (used by)
- user_validator.py (depends on)
- test_user_controller.py (tests for)

Practical SELECT Implementation

Here's how to implement selection in your workflow:

# Step 1: Create context index
context_index = {
    "files": scan_codebase(),
    "docs": parse_documentation(),
    "conversations": load_chat_history(),
    "decisions": extract_decision_log()
}

# Step 2: Define selection criteria
criteria = {
    "task": "Add password reset feature",
    "relevance_threshold": 0.7,
    "max_tokens": 50000,
    "include_tests": True
}

# Step 3: Smart selection
selected = smart_select(context_index, criteria)
# Returns: Exactly what's needed, nothing more

COMPRESS: Maximum Information, Minimum Tokens

The Compression Challenge

You've selected relevant context, but it's too large. How do you compress without losing meaning?

Compression Techniques

1. Hierarchical Summarization

Compress conversations while preserving key information:

Original Conversation (5,000 tokens):
User: "I need to add user authentication..."
Claude: "I'll help you implement authentication..."
[... 50 more exchanges ...]

Compressed Version (500 tokens):
## Authentication Implementation Discussion
- Decided: JWT tokens over sessions
- Architecture: Middleware-based validation
- Key files: auth.py, middleware.py, user_model.py
- Pending: Email verification flow
- Blockers: Need SMTP configuration

2. Structured Extraction

Convert verbose content to structured data:

# Before (verbose): 847 tokens
"""
In our meeting, we discussed that the API should 
handle errors gracefully. John suggested using 
standard HTTP status codes. Mary wanted custom 
error messages. We agreed on a hybrid approach...
"""

# After (structured): 124 tokens
{
    "decisions": {
        "error_handling": "HTTP codes + custom messages",
        "authentication": "JWT with refresh tokens",
        "rate_limiting": "100 requests/minute"
    },
    "action_items": ["Implement error middleware"],
    "participants": ["John", "Mary"]
}

3. Delta Compression

Only include what changed:

# Instead of full file content
# Track changes since last context
file_delta = {
    "auth.py": {
        "added_functions": ["reset_password", "send_reset_email"],
        "modified_functions": ["validate_token"],
        "deleted_functions": [],
        "line_changes": "+47, -12"
    }
}

Compression Best Practices

Preserve Decision Rationale
- Keep the "why" not just the "what"
- Compress implementation details, not reasoning
Maintain Searchability
- Keep key terms and identifiers
- Compress descriptions, not names
Progressive Detail
- High-level summary first
- Details available on demand

ISOLATE: Preventing Context Pollution

The Pollution Problem

When contexts mix, AI gets confused:

Mixed Context (Bad):
- Authentication code
- Marketing copy
- Database schemas
- UI components
- Customer support logs
Result: AI suggestions blend concerns inappropriately

Isolation Strategies

1. Domain-Specific Agents

Create specialized agents for different concerns:

# Agent Registry
agents = {
    "backend_dev": {
        "context": ["api/", "models/", "tests/"],
        "expertise": "Python, Django, REST APIs",
        "memory": "backend_conversations.db"
    },
    "frontend_dev": {
        "context": ["src/components/", "styles/"],
        "expertise": "React, TypeScript, CSS",
        "memory": "frontend_conversations.db"
    },
    "devops": {
        "context": ["docker/", ".github/", "scripts/"],
        "expertise": "Docker, K8s, CI/CD",
        "memory": "devops_conversations.db"
    }
}

2. Context Boundaries

Establish clear boundaries:

class ContextBoundary:
    def __init__(self, domain):
        self.domain = domain
        self.allowed_paths = self._get_allowed_paths()
        self.forbidden_contexts = self._get_forbidden()
    
    def filter_context(self, requested_context):
        # Only return context within boundaries
        return [
            ctx for ctx in requested_context
            if self._is_allowed(ctx)
        ]

3. Temporal Isolation

Separate past and present:

# Don't mix old architectural decisions with current code
contexts = {
    "historical": {
        "period": "before_refactor_2024",
        "purpose": "understanding evolution",
        "read_only": True
    },
    "current": {
        "period": "after_refactor_2024",
        "purpose": "active development",
        "read_write": True
    }
}

Multi-Agent Coordination

When agents need to work together:

# Handoff Protocol
def agent_handoff(from_agent, to_agent, context):
    summary = from_agent.summarize_work()
    relevant_context = from_agent.extract_relevant_for(to_agent)
    
    handoff_package = {
        "summary": summary,
        "context": relevant_context,
        "open_questions": from_agent.get_open_questions(),
        "constraints": from_agent.get_constraints()
    }
    
    to_agent.receive_handoff(handoff_package)

COMPOSE: Orchestrating Multiple Contexts

The Composition Challenge

Real problems require multiple contexts working together:

Code context + Documentation context
Current state + Historical decisions
User requirements + Technical constraints

Composition Patterns

1. Layered Architecture

Stack contexts from general to specific:

context_stack = [
    SystemContext(       # Bottom layer: General rules
        "coding_standards.md",
        "architecture_decisions.md"
    ),
    ProjectContext(      # Middle layer: Project-specific
        "project_readme.md",
        "api_spec.yaml",
        "database_schema.sql"
    ),
    TaskContext(         # Top layer: Current task
        "current_feature.md",
        "related_code_files.py",
        "recent_conversations.json"
    )
]

2. Context Weaving

Intelligently merge related contexts:

def weave_contexts(contexts):
    # Identify overlaps
    overlap_map = find_context_overlaps(contexts)
    
    # Merge related information
    woven = {}
    for topic in overlap_map:
        woven[topic] = merge_perspectives(
            contexts, 
            topic,
            preserve_contradictions=True
        )
    
    return woven

# Example output:
{
    "user_authentication": {
        "code_perspective": "JWT implementation in auth.py",
        "doc_perspective": "Security best practices guide",
        "test_perspective": "Auth test coverage at 87%",
        "decision_history": "Moved from sessions (2023) to JWT (2024)"
    }
}

3. Dynamic Composition

Adjust composition based on task:

class DynamicComposer:
    def compose_for_task(self, task_type):
        if task_type == "bug_fix":
            return self._compose_debugging_context()
        elif task_type == "new_feature":
            return self._compose_creative_context()
        elif task_type == "refactoring":
            return self._compose_architectural_context()
    
    def _compose_debugging_context(self):
        return {
            "priority": ["error_logs", "recent_changes", "test_results"],
            "include": ["stack_traces", "related_issues", "system_state"],
            "exclude": ["future_plans", "unrelated_features"]
        }

Practical SCIC Implementation

Complete Example: Code Review System

Let's build a context-aware code review system using SCIC:

class ContextAwareReviewer:
    def __init__(self):
        self.selector = ContextSelector()
        self.compressor = ContextCompressor()
        self.isolator = ContextIsolator()
        self.composer = ContextComposer()
    
    def review_pull_request(self, pr_id):
        # SELECT: Gather relevant context
        selected = self.selector.select({
            "changed_files": self.get_pr_files(pr_id),
            "related_files": self.find_dependencies(),
            "test_files": self.find_related_tests(),
            "past_reviews": self.get_similar_reviews(),
            "team_standards": self.load_coding_standards()
        })
        
        # COMPRESS: Reduce to essentials
        compressed = self.compressor.compress({
            "summarize_changes": self.summarize_diff(selected),
            "extract_patterns": self.identify_patterns(selected),
            "compress_history": self.compress_review_history(selected)
        })
        
        # ISOLATE: Separate concerns
        isolated = self.isolator.isolate({
            "security_context": self.security_focused_context(compressed),
            "performance_context": self.performance_focused_context(compressed),
            "style_context": self.style_focused_context(compressed)
        })
        
        # COMPOSE: Create unified review
        review = self.composer.compose({
            "integrate_perspectives": isolated,
            "prioritize_issues": self.rank_by_severity(),
            "generate_suggestions": self.create_improvements()
        })
        
        return review

Hands-on Exercise

Build Your Own SCIC System

Create a simple SCIC implementation for your use case:

# Template for your SCIC system
class MySCICSystem:
    def __init__(self, domain):
        self.domain = domain
        
    def select(self, query):
        # TODO: Implement your selection logic
        pass
    
    def compress(self, selected_context):
        # TODO: Implement your compression
        pass
    
    def isolate(self, compressed_context):
        # TODO: Implement your isolation
        pass
    
    def compose(self, isolated_contexts):
        # TODO: Implement your composition
        pass
    
    def process(self, task):
        selected = self.select(task)
        compressed = self.compress(selected)
        isolated = self.isolate(compressed)
        result = self.compose(isolated)
        return result

Implementation Checklist

SELECT: Implement at least 2 selection strategies
COMPRESS: Reduce context by at least 70%
ISOLATE: Create 3 separate context domains
COMPOSE: Merge contexts without conflicts

Checkpoint Task

Your Mission

Implement a simple SCIC system that maintains context across 5 conversation turns:

Choose a scenario (e.g., helping plan a project)
Implement each SCIC component:
- SELECT: Choose what matters for each turn
- COMPRESS: Summarize previous turns
- ISOLATE: Keep different aspects separate
- COMPOSE: Bring it together coherently
Test with 5 turns:
- Turn 1: Initial request
- Turn 2: Clarification
- Turn 3: Additional requirements
- Turn 4: Change in direction
- Turn 5: Final integration
Measure success:
- Context stays under token limit
- No loss of critical information
- AI maintains coherent understanding
- Responses improve with each turn