Skip to main content

Source: ocean/docs/slack-automation-setup.md | ✏️ Edit on GitHub

Slack Automation Setup Guide

This guide covers setting up Slack for automated notifications from Sentry, PostHog, and custom alerts.

Table of Contents​

  1. Creating a Slack App
  2. Setting up Incoming Webhooks
  3. Sentry Integration
  4. PostHog Integration
  5. Custom Alerts
  6. Security Best Practices

Creating a Slack App​

Step 1: Create a New Slack App​

  1. Go to API.slack.com/apps
  2. Click "Create New App"
  3. Choose "From scratch"
  4. Enter app details:
    • App Name: "Ocean Monitoring" (or your preferred name)
    • Workspace: Select your workspace
  5. Click "Create App"

Step 2: Configure App Permissions​

  1. In your app settings, go to "OAuth & Permissions"

  2. Under "Bot Token Scopes", add:

    • chat:write - Post messages
    • chat:write.public - Post to public channels
    • files:write - Upload files (for error screenshots)
    • incoming-webhook - Use incoming webhooks

  3. Click "Install to Workspace"

  4. Authorize the app

  5. Copy the Bot User OAuth Token (starts with xoxb-)

Step 3: Create Notification Channels​

Create dedicated Slack channels for different alert types:

#alerts-critical    - Production errors, downtime
#alerts-errors      - Application errors from Sentry
#alerts-performance - Performance issues, slow queries
#alerts-security    - Security events, failed auth attempts
#alerts-analytics   - PostHog events, user behavior anomalies

Setting up Incoming Webhooks​

  1. In your Slack app settings, go to "Incoming Webhooks"
  2. Toggle "Activate Incoming Webhooks" to On
  3. Click "Add New Webhook to Workspace"
  4. Select the channel (e.g., #alerts-errors)
  5. Copy the webhook URL

Option 2: Legacy Webhooks (Simple but Limited)​

  1. Go to your Slack workspace settings
  2. Navigate to "Apps" → "Custom Integrations" → "Incoming Webhooks"
  3. Click "Add to Slack"
  4. Choose a channel and click "Add Incoming Webhooks Integration"
  5. Copy the webhook URL

Sentry Integration​

  1. In Sentry, go to Settings → Integrations
  2. Search for "Slack" and click "Install"
  3. Click "Add Workspace"
  4. Authorize Sentry to access your Slack workspace
  5. Configure alert rules in Alerts → Create Alert Rule

Method 2: Webhook Integration​

  1. In Sentry, go to Settings → Integrations → Webhooks
  2. Click "Add Webhook"
  3. Enter your Slack webhook URL
  4. Configure payload (Sentry will send JSON automatically)

Configuring Sentry Alert Rules for Slack​

  1. Go to Alerts → Alert Rules
  2. Click "Create Alert Rule"
  3. Choose alert type:
    • Issue Alert: For new or regression errors
    • Metric Alert: For performance, error rate thresholds

Example: Critical Error Alert​

Alert Name: 'Critical Production Errors'
Environment: Production
Conditions:
  - When: An event is seen
  - Filter:
      - Level = Error or Fatal
      - Environment = production
      - Tags: alert_type = critical OR pg_net_status = critical
Actions:
  - Send a Slack notification to #alerts-critical
  - Include: Error message, User context, Stack trace preview

Example: Performance Alert​

Alert Name: 'Slow Database Queries'
Environment: All
Conditions:
  - When: p95(transaction.duration) > 3000ms
  - For: 5 minutes
Actions:
  - Send a Slack notification to #alerts-performance
  - Include: Transaction name, Duration, Affected users

Example: Error Rate Spike​

Alert Name: 'High Error Rate'
Conditions:
  - When: Error rate > 5%
  - For: 10 minutes
  - Compared to: 1 hour ago
Actions:
  - Send a Slack notification to #alerts-errors
  - Trigger PagerDuty (if configured)

PostHog Integration​

Setting up PostHog Webhooks​

  1. In PostHog, go to Project Settings → Webhooks
  2. Click "New Webhook"
  3. Configure webhook:
    • URL: Your Slack webhook URL
    • Events: Choose which events to send
    • Format: Slack-compatible JSON

Creating PostHog Actions for Slack​

  1. Go to Data Management → Actions

  2. Click "New Action"

  3. Define the action (e.g., "User Signup")

  4. Go to Apps → Browse Apps

  5. Install "Slack Webhook" app

  6. Configure:

    {
      "webhook_url": "YOUR_SLACK_WEBHOOK_URL",
      "message_format": "New user signed up: {person.email}",
      "channel": "#alerts-analytics"
    }

PostHog Alert Examples​

User Behavior Alerts​

// High cart abandonment rate
if (event === 'cart_abandoned' && properties.cart_value > 100) {
  webhook.send({
    text: `High-value cart abandoned: $${properties.cart_value}`,
    channel: '#alerts-analytics',
  })
}

Feature Flag Alerts​

// Feature flag error rate
if (event === 'feature_flag_error' && properties.flag_key === 'new_checkout') {
  webhook.send({
    text: `Feature flag "${properties.flag_key}" causing errors`,
    channel: '#alerts-critical',
  })
}

Custom Alerts​

Creating a Slack Webhook Handler​

Save your Slack webhook URLs in environment variables:

VITE_SLACK_WEBHOOK_CRITICAL=https://hooks.slack.com/services/xxx/yyy/zzz
VITE_SLACK_WEBHOOK_ERRORS=https://hooks.slack.com/services/xxx/yyy/zzz
VITE_SLACK_WEBHOOK_PERFORMANCE=https://hooks.slack.com/services/xxx/yyy/zzz

Edge Function for Slack Notifications​

Create a Supabase Edge Function to handle Slack notifications:

// supabase/functions/send-slack-alert/index.ts
import { serve } from 'https://deno.land/std@0.208.0/http/server.ts'

const SLACK_WEBHOOKS = {
  critical: Deno.env.get('SLACK_WEBHOOK_CRITICAL')!,
  error: Deno.env.get('SLACK_WEBHOOK_ERRORS')!,
  performance: Deno.env.get('SLACK_WEBHOOK_PERFORMANCE')!,
  security: Deno.env.get('SLACK_WEBHOOK_SECURITY')!,
  analytics: Deno.env.get('SLACK_WEBHOOK_ANALYTICS')!,
}

interface SlackMessage {
  text: string
  channel?: string
  username?: string
  icon_emoji?: string
  attachments?: Array<{
    color: 'danger' | 'warning' | 'good' | string
    title: string
    text?: string
    fields?: Array<{
      title: string
      value: string
      short?: boolean
    }>
    footer?: string
    ts?: number
  }>
}

serve(async (req) => {
  try {
    const { type, level, message, details } = await req.json()

    const webhook = SLACK_WEBHOOKS[level] || SLACK_WEBHOOKS.error

    const slackMessage: SlackMessage = {
      text: message,
      username: 'Ocean Monitoring',
      icon_emoji: ':ocean:',
      attachments: [
        {
          color: level === 'critical' ? 'danger' : level === 'warning' ? 'warning' : 'good',
          title: `${type} Alert`,
          fields: Object.entries(details || {}).map(([key, value]) => ({
            title: key.replace(/_/g, ' ').replace(/\b\w/g, (l) => l.toUpperCase()),
            value: String(value),
            short: true,
          })),
          footer: 'Ocean Monitoring',
          ts: Date.now() / 1000,
        },
      ],
    }

    const response = await fetch(webhook, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(slackMessage),
    })

    if (!response.ok) {
      throw new Error(`Slack API error: ${response.statusText}`)
    }

    return new Response(JSON.stringify({ success: true }), {
      headers: { 'Content-Type': 'application/json' },
    })
  } catch (error) {
    return new Response(JSON.stringify({ error: error.message }), {
      status: 500,
      headers: { 'Content-Type': 'application/json' },
    })
  }
})

Database Trigger for Critical Alerts​

-- Send Slack alert when pg_net queue is critical
CREATE OR REPLACE FUNCTION monitoring.send_slack_alert_on_critical()
RETURNS trigger AS $$
BEGIN
  IF NEW.status = 'critical' AND NEW.queue_size > 10000 THEN
    -- Call Edge Function to send Slack alert
    PERFORM net.http_post(
      url := 'https://YOUR_PROJECT.supabase.co/functions/v1/send-slack-alert',
      headers := jsonb_build_object(
        'Content-Type', 'application/json',
        'Authorization', 'Bearer ' || current_setting('app.service_role_key')
      ),
      body := jsonb_build_object(
        'type', 'pg_net Health',
        'level', 'critical',
        'message', format('CRITICAL: pg_net backlog detected - %s items in queue', NEW.queue_size),
        'details', jsonb_build_object(
          'queue_size', NEW.queue_size,
          'response_size', NEW.response_size,
          'timestamp', NEW.created_at
        )
      )
    );
  END IF;

  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER send_slack_on_pg_net_critical
  AFTER INSERT ON monitoring.pg_net_health
  FOR EACH ROW
  EXECUTE FUNCTION monitoring.send_slack_alert_on_critical();

Security Best Practices​

1. Webhook URL Protection​

  • Never commit webhook URLs to git
  • Store in environment variables or secure vault
  • Rotate webhook URLs periodically
  • Use different webhooks for different severity levels

2. Rate Limiting​

Implement rate limiting to prevent alert storms:

const rateLimiter = new Map<string, number>()
const RATE_LIMIT_WINDOW = 60000 // 1 minute
const MAX_ALERTS_PER_WINDOW = 10

function shouldSendAlert(alertKey: string): boolean {
  const now = Date.now()
  const lastSent = rateLimiter.get(alertKey) || 0

  if (now - lastSent < RATE_LIMIT_WINDOW) {
    return false
  }

  rateLimiter.set(alertKey, now)
  return true
}

3. Message Sanitization​

Always sanitize user input in Slack messages:

function sanitizeForSlack(text: string): string {
  return text.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;')
}

4. Channel Access Control​

  • Create dedicated channels for alerts
  • Limit who can modify channel settings
  • Use channel naming conventions (e.g., #alerts-*)
  • Archive old alert channels periodically

Testing Your Integration​

1. Test Webhook​

curl -X POST -H 'Content-Type: application/json' \
  -d '{"text":"Test alert from Ocean monitoring"}' \
  YOUR_SLACK_WEBHOOK_URL

2. Test Sentry Integration​

// Trigger a test error in your app
Sentry.captureMessage('Test Slack integration', {
  level: 'error',
  tags: {
    test: true,
    channel: 'alerts-errors',
  },
})

3. Test PostHog Integration​

// Trigger a test event
posthog.capture('test_slack_integration', {
  alert_type: 'test',
  channel: 'alerts-analytics',
})

Troubleshooting​

Common Issues​

  1. "Invalid webhook URL"

    • Verify URL format: https://hooks.slack.com/services/xxx/yyy/zzz
    • Check if webhook is still active in Slack

  2. "No messages appearing"

    • Check channel permissions
    • Verify webhook is configured for correct channel
    • Check Slack app permissions

  3. "Rate limited"

    • Implement rate limiting in your code
    • Use message threading for related alerts
    • Batch similar alerts

Debug Logging​

Enable debug logging for Slack integrations:

if (process.env.NODE_ENV === 'development') {
  console.log('Slack webhook payload:', JSON.stringify(slackMessage, null, 2))
}

Next Steps​

  1. Set up alert aggregation to prevent spam
  2. Create custom alert templates for different error types
  3. Implement alert acknowledgment workflow
  4. Set up on-call rotation with PagerDuty integration
  5. Create Slack slash commands for quick actions