Log Analysis Quiz

Two-Pass Processing Pattern:

A two-pass processing pattern separates analysis from mutation by processing the dataset twice.

Pass 1: Analyze (read-only)

• Scan the data to detect patterns or conflicts and build the required state (e.g., find duplicate UIDs and create a reassignment map).

Pass 2: Apply (write/action)

• Apply corrections using only the results from Pass 1, without making new decisions.

When to use: Use this pattern when fixes depend on global knowledge of the data, and modifying records safely requires seeing the full dataset first. Example: Finding duplicate UIDs and reassigning them to available unique values.

find()

• Returns -1 if substring not found
• Silent failure - can cause subtle bugs with slicing
• Use when you want to check and handle missing substrings with conditionals

index()

• Raises ValueError if substring not found
• Explicit error handling with try/except
• Clearer for error cases

Best practice: Use index() with try/except for clearer error handling in log parsing.

Three options with trade-offs:

dict with .get() - Simple counting, minimal overhead

count[item] = count.get(item, 0) + 1

defaultdict(int) - Cleaner code, no .get() needed

count[item] += 1

Counter - When you need .most_common() or count arithmetic

Choose based on needs: Start simple (dict), use defaultdict for cleaner code, use Counter when you need its special features.

set - Ideal for uniqueness tracking

Benefits:

• O(1) membership testing: if item in seen_set
• Automatic deduplication
• Memory efficient for large unique sets

Example use cases: Track unique IP addresses, pod names, user IDs

Alternative: dict keys work but are overkill unless you need associated values.

defaultdict(list) - Purpose-built for grouping

Why it’s ideal:

• Auto-initializes missing keys to empty lists
• No existence checks needed: pod_events[pod_name].append(event)
• Clean, readable code

Manual alternatives (from explicit to Pythonic):

# Most explicit - shows the logic clearly
if key in my_dict:
    my_dict[key].append(value)
else:
    my_dict[key] = [value]

# More concise with setdefault
dict.setdefault(key, []).append(value)

Use Counter when you need:

• .most_common(N): Find top N most common items
• Count arithmetic: Combine counts from multiple sources (count1 + count2)
• Multiple count operations: Subtract, intersect, union

Use dict/defaultdict when:

• Simple counting without special operations
• Want minimal overhead
• Don’t need Counter’s features

Key insight: Counter is a specialized tool - use it when you need its features, not just for basic counting.

Delimited Files (CSV-like):

• Use split(’:’) or split(’,')
• Fixed column positions
• Flat structure
• Must handle comments (#) and empty lines manually
• Simple index-based access: data[2]

JSON Logs:

• Use json.loads() per line
• Named fields with .get()
• Nested structures
• Type-safe (booleans, numbers preserved)
• Safer with .get() defaults: event.get(‘field’, {})

Key insight: JSON is self-describing and handles nesting; delimited is simpler but requires knowing column positions.

Early Continue Pattern:

Early continue enables fail-fast logic, keeping the main logic flat, readable, and focused on valid cases.

for event in events:
    if obj.get('kind') != 'Pod':
        continue
    if event.get('type') != 'Warning':
        continue
    # Process filtered event

Benefits:

• Improves readability with many conditions
• Reduces nesting levels
• Makes filtering logic explicit
• Each condition is independent and clear

Alternative: Chain with ‘and’ for compact code with few conditions

Best practice: Use early continue when you have 3+ filtering conditions.

String sort is sufficient when:

• You only need chronological ordering
• All timestamps share the same consistent ISO 8601-like format (e.g., 2025-08-12T09:05:00)
• Lexicographic order == chronological order ✓

Convert to datetime when:

• You need duration math: logout_dt - login_dt
• You need to add/subtract time: ts + timedelta(hours=1)
• You need to compare across different formats

Common pattern:

# Step 1 — sort cheaply as strings
sorted_events = sorted(events, key=lambda e: e['timestamp'])

# Step 2 — convert only for duration calculation
from datetime import datetime
login_dt  = datetime.fromisoformat(login['timestamp'])
logout_dt = datetime.fromisoformat(logout['timestamp'])
session_duration = logout_dt - login_dt

Key insight: Don’t pay the cost of datetime conversion unless you need arithmetic.