Skip to main content

Source: ocean/docs/adr/014-cloud-first-testing-strategy.md | ✏️ Edit on GitHub

ADR-014: Cloud-First Testing Strategy

Status​

Accepted

Context​

Our development approach is cloud-first, using remote Supabase instances and cloud services rather than local development environments. However, our test suite was built with assumptions about local services (local Supabase, localhost:54321) that don't align with this approach.

Problems with the current testing approach:

  • Tests fail when local Supabase isn't running
  • Mock environments don't catch real integration issues
  • Local testing can't replicate cloud-specific problems (rate limits, auth flows, network latency)
  • Tests that run full builds are resource-intensive and slow
  • Environment-specific bugs only appear in production

Decision​

We are adopting a cloud-first testing strategy that aligns with our cloud-first development approach:

1. Test Categories​

Keep Locally (Fast, Pure Tests)

  • Static code analysis (security checks, code quality)
  • Pure function tests (no external dependencies)
  • Component logic tests with properly mocked dependencies

Move to CI/CD (Integration Tests)

  • Tests requiring real services (Supabase, Stripe, etc.)
  • Build validation and bundle size checks
  • E2E tests against preview deployments
  • Smoke tests after deployments

2. Testing Infrastructure​

Local Development

  • Pre-commit hooks run only static analysis
  • Unit tests mock at the module boundary
  • No dependency on local services

CI/CD Pipeline

  • GitHub Actions run integration tests against test Supabase project
  • Preview deployments get full E2E test suite
  • Production deployments trigger smoke tests
  • Synthetic monitoring for continuous validation

3. Implementation Changes​

Removed Tests

  • tests/graphql-signup.test.ts - Required Supabase instance
  • tests/pre-push/build-integrity.test.ts - Ran full build process
  • Exec calls in deployment-readiness.test.ts - Ran lint/format/typecheck

Modified Tests

  • JWT verifier test - Mock at module boundary to avoid URL creation
  • Deployment readiness - Keep static checks, remove dynamic validations

New Approach

  • Contract testing with recorded API responses
  • Feature flag testing in production with PostHog
  • Synthetic monitoring with real user flows

Consequences​

Positive​

  • Tests align with actual infrastructure
  • Catch real integration issues early
  • Faster local development (no service dependencies)
  • More reliable test results
  • Better coverage of cloud-specific scenarios

Negative​

  • Can't run full test suite locally
  • Dependency on CI/CD for integration testing
  • Need proper test data management in cloud
  • Potential for higher cloud service costs

Mitigation​

  • Use GitHub Codespaces for full environment testing
  • Implement proper test data cleanup
  • Monitor test environment costs
  • Use Supabase branching for isolated test environments

Implementation Guidelines​

For Developers​

  1. Writing Tests

    • Mock external services at the module boundary
    • No hardcoded URLs or localhost references
    • Use environment variables for configuration

  2. Running Tests

    # Local - only unit tests
    pnpm test

    # CI/CD - full suite including integration
    # Automatically run on PR

  3. Test Data

    • Use unique identifiers for test data
    • Implement cleanup in test teardown
    • Never use production data

For CI/CD​

  1. PR Checks

    • Static analysis and unit tests
    • Deploy preview with Supabase branch
    • Run E2E tests against preview

  2. Main Branch

    • All PR checks plus integration tests
    • Deploy to staging (when available)
    • Run smoke tests

  3. Production

    • Synthetic monitoring
    • Feature flag testing
    • Real user monitoring

Tools and Services​

  • Local: Vitest for unit tests
  • E2E: Playwright (future)
  • Monitoring: Checkly or Datadog Synthetics (future)
  • API Testing: MSW for contract testing (future)
  • Feature Flags: PostHog

Migration Path​

  1. ✅ Remove tests requiring local services
  2. ✅ Update remaining tests to mock properly
  3. ⏳ Add E2E tests in CI/CD pipeline
  4. ⏳ Implement synthetic monitoring
  5. ⏳ Add contract testing for APIs

References​