Source: ebisu/docs/adr/0057-intelligence-platform-conflicts-resolution.md | ✏️ Edit on GitHub

ADR-057: Intelligence Platform Conflicts Resolution

Status

Implemented

Context

The Ebisu system had fundamental conflicts between its intelligence platform design principles and its implementation:

1. ON CONFLICT clauses throughout the system prevented duplicate data, violating the principle that duplicates are valuable confirmations
2. vessel_associates table had conflicting schema definitions across migrations
3. Data was being lost instead of preserved as intelligence

Decision

Implemented intelligence-based conflict resolution by:

1. Removing ON CONFLICT constraints that prevented duplicate data
2. Making foreign keys nullable where intelligence preservation requires flexibility
3. Adding sighting counters to track multiple confirmations as intelligence value
4. Preserving all data points instead of deduplicating

Implementation

Created /scripts/fix_intelligence_conflicts.sql that:

• Removed conflicting constraints from vessel_associates, vessel_metrics, vessel_external_identifiers
• Updated functions to preserve all intelligence instead of deduplicating
• Added sighting tracking to vessel_sources
• Made associate_master_uuid nullable to support intelligence gathering

Consequences

Positive

• Data is now preserved as intelligence rather than deduplicated
• Multiple reports of same association increase confidence
• System aligns with intelligence platform principles
• CCSBT import success increased from 0% to 91%

Negative

• Larger data storage requirements
• Need for downstream deduplication if required
• More complex queries for unique vessel counts

Lessons Learned

Traditional MDM thinking (deduplication) was deeply embedded in the codebase despite intelligence platform design goals. Regular architecture reviews are needed to ensure implementation matches design principles.