Source: ebisu/docs/adr/0057-missing-data-sources-required.md | ✏️ Edit on GitHub

ADR-0057: Missing Data Sources Required for Complete Intelligence Platform

Status

Accepted

Context

Our maritime intelligence platform has successfully imported vessel data from 12 sources (~45,000 vessel records), but critical data sources are missing. These missing sources are essential for:

• Comprehensive risk assessment
• Complete beneficial ownership tracking
• IUU fishing detection
• Sanctions compliance
• Flag state verification

Decision

Document all missing data sources that must be obtained before proceeding to Phase 2 (Cross-Source Identity Resolution). These sources are already defined in original_sources_vessels but lack actual data files.

Critical Missing Data Sources

1. IUU Vessel Lists (Highest Priority)

These identify vessels engaged in illegal, unreported, and unregulated fishing:

Source	Description	Typical Format	Public Access
CCAMLR_IUU	Commission for Conservation of Antarctic Marine Living Resources	PDF/Web	Yes - ccamlr.org
CCSBT_IUU	Commission for Conservation of Southern Bluefin Tuna	PDF/Excel	Yes - ccsbt.org
GFCM_IUU	General Fisheries Commission for the Mediterranean	Web/PDF	Yes - fao.org/gfcm
IATTC_IUU	Inter-American Tropical Tuna Commission	PDF/Web	Yes - iattc.org
ICCAT_IUU	International Commission for Conservation of Atlantic Tunas	Excel/Web	Yes - iccat.int
IOTC_IUU	Indian Ocean Tuna Commission	Excel/PDF	Yes - iotc.org
NAFO_IUU	Northwest Atlantic Fisheries Organization	Web	Yes - nafo.int
NEAFC_IUU	North East Atlantic Fisheries Commission	Web	Yes - neafc.org
NPFC_IUU	North Pacific Fisheries Commission	PDF/Web	Yes - npfc.int
SEAFO_IUU	South East Atlantic Fisheries Organisation	PDF	Yes - seafo.org
SIOFA_IUU	Southern Indian Ocean Fisheries Agreement	Web	Yes - siofa.org
SPRFMO_IUU	South Pacific RFMO	Web/Excel	Yes - sprfmo.int
WCPFC_IUU	Western & Central Pacific Fisheries Commission	PDF/Web	Yes - wcpfc.int

2. Missing RFMO Authorized Vessel Lists

Source	Description	Status
GFCM	General Fisheries Commission for the Mediterranean	No data file
SEAFO	South East Atlantic Fisheries Organisation	Have PDF, need extraction
SIOFA	Southern Indian Ocean Fisheries Agreement	No data file
CCAMLR	Antarctic Marine Living Resources	No data file

3. Country Fleet Registers (Flag State Verification)

European Union Fleet Register:

• 29 EU member states (EU_BEL through EU_SWE)
• Available at: ec.europa.eu/fisheries/fleet
• Format: CSV/Excel export
• Contains: CFR number, IMO, vessel details, ownership

Other National Registers:

Country	Source	Access	Key Value
Norway	NOR_VESSELS	fiskeridir.no	Major fishing nation
UK	GBR_LARGE, GBR_SMALL	gov.uk	Post-Brexit fleet
Mexico	MEX_LARGE, MEX_SMALL	conapesca.gob.mx	Large Pacific fleet
Faroe Islands	FRO_VESSELS	skipaskra.fo	Significant Atlantic fleet
Russia	RUS_VESSELS	fish.gov.ru	Major distant water fleet
Taiwan	TWN_PAC, TWN_CAR_SIOFA, TWN_FV_SIOFA	fa.gov.tw	Large tuna fleet
Panama	PAN_VESSELS	arap.gob.pa	Major flag state
Maldives	MDV_VESSELS	fishagri.gov.mv	Indian Ocean fleet
USA Alaska	USA_AK	adfg.alaska.gov	Pacific fisheries

4. Civil Society Sources (Sustainability & Compliance)

ISSF (International Seafood Sustainability Foundation):

• ISSF_PS: Large-Scale Purse Seine Vessels
• ISSF_PVR: ProActive Vessel Register (best practices)
• ISSF_UVI: UVI/IMO Vessel List
• ISSF_VOSI: Vessels in Other Sustainability Initiatives
• Available at: iss-foundation.org
• Format: Excel/CSV

MSC (Marine Stewardship Council):

• MSC_VESSELS: Vessels in certified fisheries
• Available at: msc.org
• Format: Via API or fishery certificates

Others:

• AP2HI: Indonesian tuna association registry
• OUTLAW_OCEAN: Investigative journalism vessel database

5. Intergovernmental Sources

PNA (Parties to the Nauru Agreement):

• PNA_FSMA: Federated States of Micronesia Arrangement
• PNA_TUNA: Vessel Day Scheme registry
• Available at: pnatuna.com
• Critical for Pacific tuna management

Data Acquisition Strategy

1. Automated Collection (where possible):

   • Write scrapers for web-based IUU lists
   • Use APIs where available (EU fleet, MSC)
   • Set up periodic updates

2. Manual Collection (where necessary):

   • Download PDFs and extract data
   • Contact RFMOs directly for machine-readable formats
   • Establish data sharing agreements

3. Priority Order:

   • IUU lists (critical for risk assessment)
   • Major flag state registers (EU, Norway, Taiwan)
   • RFMO gaps (GFCM, SIOFA, CCAMLR)
   • Civil society sources

Technical Requirements

1. Data Extractors Needed:

   • PDF parser for SEAFO and other PDF-only sources
   • Web scraper for online IUU lists
   • Excel/CSV processors with format detection

2. Import Scripts:

   • Standardized cleaning scripts for each source type
   • Staged import scripts following existing patterns
   • Data quality validation

3. Update Mechanisms:

   • Track last update date for each source
   • Automated checks for new versions
   • Change detection and incremental updates

Consequences

Positive

• Complete global vessel coverage for risk assessment
• Comprehensive IUU detection across all ocean basins
• Verified flag state data for ownership tracking
• Industry sustainability certifications included

Negative

• Significant effort required for data collection
• Ongoing maintenance for updates
• Some sources may require manual processing
• Data formats vary widely

Neutral

• Increases data volume by ~50-100K vessels
• More complex matching in Phase 2
• Higher infrastructure requirements

Implementation Notes

1. Before Phase 2:

   • Must have at least IUU lists
   • Should have major flag states (EU, Norway, Taiwan)
   • Nice to have civil society sources

2. Data Quality:

   • Each source needs custom validation
   • Standardize to common schema
   • Preserve source-specific fields

3. Legal Considerations:

   • All listed sources are publicly available
   • Respect terms of use
   • Maintain attribution

Next Steps

1. Create data collection scripts for IUU lists
2. Contact RFMOs for machine-readable formats
3. Set up automated EU fleet register downloads
4. Establish update schedule for each source