Skip to main content

Source: ebisu/docs/IMPORT_MIGRATION_GUIDE.md | ✏️ Edit on GitHub

Import System Migration Guide

Overview​

This guide helps migrate from the old scattered import system to the new organized structure.

What Changed​

Old System​

  • 60+ scripts in scripts/import/vessels/data/
  • Data files mixed with code in Git
  • Hardcoded database credentials
  • Inconsistent patterns

New System​

  • Organized scripts in import-sources/
  • Data files tracked with Git LFS
  • Environment-based credentials
  • Unified import interface

Migration Steps​

1. Set Up Environment​

# Create secure environment file
cp .env.example .env
./setup-env.sh

# Source it before working
source .env

2. Initialize Git LFS​

# One-time setup
./setup-git-lfs.sh

# Verify it's working
git lfs ls-files

3. Migrate Existing Data Files​

If you have uncommitted data files:

# Option A: Add to Git LFS
git add import/vessels/vessel_data/**/*.csv
git commit -m "feat: add vessel data files to Git LFS"

# Option B: Use local staging
./scripts/migrate_data_files.sh

4. Update Import Commands​

Old Way​

docker exec -i ebisu-importer bash -c "
  export DATABASE_URL='postgresql://user:pass@host:5432/db' &&
  /app/scripts/import/vessels/data/COUNTRY/load_usa_ak_vessels.sh"

New Way​

source .env
./import-sources/docker-import.sh country/usa-alaska

Common Import Examples​

Country Imports​

# USA Alaska
./import-sources/docker-import.sh country/usa-alaska

# Chile LTP-PEP
./import-sources/docker-import.sh country/chile-ltp-pep

# EU Countries (requires country code)
./import-sources/docker-import.sh country/eu-fleet ESP
./import-sources/docker-import.sh country/eu-fleet FRA
./import-sources/docker-import.sh country/eu-fleet ITA

RFMO Imports​

# All major RFMOs
./import-sources/docker-import.sh rfmo/iccat
./import-sources/docker-import.sh rfmo/iotc
./import-sources/docker-import.sh rfmo/wcpfc
./import-sources/docker-import.sh rfmo/nafo

Adding New Data​

For Version Control (Git LFS)​

  1. Place file in correct directory:

    cp new_data.csv import/vessels/vessel_data/COUNTRY/USA_AK/raw/

  2. Add to Git LFS:

    git add import/vessels/vessel_data/COUNTRY/USA_AK/raw/new_data.csv
    git commit -m "feat: add Alaska vessel data for 2025-02"

For Local Testing​

# Add to local staging
./import-sources/manage-data.sh add "COUNTRY/USA_AK" test_data.csv

# Import from staging
./import-sources/docker-import.sh country/usa-alaska

Troubleshooting​

"POSTGRES_PASSWORD not set"​

  • Run: source .env
  • Verify: echo $POSTGRES_PASSWORD

"No data file found"​

  • Check both locations:
    • Git LFS: ls import/vessels/vessel_data/
    • Local: ./import-sources/manage-data.sh list

"Git LFS not initialized"​

  • Run: ./setup-git-lfs.sh
  • Verify: git lfs version

Benefits of New System​

  1. Security: No hardcoded credentials
  2. Organization: Clear directory structure
  3. Efficiency: Git LFS prevents repository bloat
  4. Consistency: Same process for all sources
  5. Simplicity: One command to import any source

Need Help?​

  • Check logs: docker logs ebisu-importer
  • Review ADR-0065 for design decisions
  • See import-sources/README.md for detailed usage