Skip to main content

Source: ebisu/docs/vessel-mdm-trust-system.md | ✏️ Edit on GitHub

Vessel MDM Trust System

Overview​

The Ebisu database now includes a comprehensive Master Data Management (MDM) system with trust scoring and historical tracking for AI readiness.

Key Components​

1. Trust Scoring System​

Tables:

  • vessel_trust_scores - Stores calculated trust scores for each vessel
  • vessel_data_conflicts - Tracks conflicting data across sources

Scoring Formula (0.0-1.0):

  • 35% Source Authority (AUTHORITATIVE > VERIFIED > REPUTABLE > UNVERIFIED > BLACKLIST)
  • 25% Identifier Strength (IMO > IRCS+MMSI > IRCS > MMSI > Name)
  • 20% Data Completeness (number of non-null fields)
  • 10% Data Consistency (fewer conflicts = higher score)
  • 10% Temporal Relevance (recent data scores higher)

AI Training Suitability:

  • Vessels with trust_score >= 0.7 and data_completeness >= 0.6 are marked as AI-suitable
  • Blacklisted vessels are excluded from AI training

2. Historical Tracking System​

Tables:

  • vessel_historical_data - Complete JSONB snapshots of vessel data over time
  • vessel_field_history - Tracks individual field changes for stability analysis

Features:

  • Captures full vessel state at each source update
  • Calculates diffs between snapshots
  • Tracks field-level stability across sources
  • Enables temporal queries and trend analysis

3. Source Authority Levels​

Current distribution:

  • AUTHORITATIVE (48): RFMOs (ICCAT, IOTC, WCPFC, etc.), Government registries
  • BLACKLIST (15): IUU lists, sanctions lists (SDN, WRO)
  • REPUTABLE (6): MSC, ISSF certification bodies
  • UNVERIFIED (3): New or unverified sources

Usage Examples​

Calculate trust scores for all vessels​

SELECT * FROM calculate_all_vessel_trust_scores();

Get AI-ready vessels​

SELECT * FROM ai_training_vessels 
WHERE trust_score >= 0.8
ORDER BY trust_score DESC;

View vessel history​

SELECT * FROM get_vessel_history('vessel-uuid');

Analyze field stability​

SELECT * FROM analyze_vessel_field_stability('vessel-uuid');

Check vessel trust summary​

SELECT * FROM get_vessel_trust_summary('vessel-uuid');

Import Integration​

All vessel imports now:

  1. Use find_or_create_vessel_with_trust() for matching
  2. Always record vessel presence in each source
  3. Capture historical snapshots with capture_vessel_snapshot()
  4. Calculate trust scores after import

Next Steps​

  1. Test the system with ICCAT data re-import
  2. Implement remaining RFMO importers (IOTC, WCPFC, IATTC)
  3. Build trust score monitoring dashboards
  4. Create data quality reports for AI training