ETL Cost Monitor

Overview

Most ETL providers today use usage-based pricing models that are both complex and opaque. Teams often lack the ability to:

• Visualize their costs at a granular level (by connector, schema, or table)
• Forecast monthly spend based on current usage
• Get alerted when there are budget overruns, unusual usage spikes, or other unexpected cost behaviors

ETL Cost Monitor solves this by giving you complete visibility and control over your ETL costs — all from within Snowflake.

What It Does

• Tracks and visualizes current month cost, forecasted cost, and total MAR
• Breaks down usage and cost by connector, schema, and table
• Highlights top gainers and most expensive tables
• Supports budget and spike alerts via Snowflake Tasks and Notification Integrations
• All logic runs inside your Snowflake account - no data is sent externally

Post-Installation Configuration

🔓 Granting Access to Usage Data

The first time you launch the app, you will be prompted to grant SELECT access to your ETL provider’s usage table - such as fivetran_metadata.incremental_mar

📬 Alerting Setup (Optional but Recommended)

Step 1: Grant App Role Access to a User Role

GRANT APPLICATION ROLE etl_cost_monitor_app.app_public TO ROLE <your_account_role>;

Alternatively, manage this via the Access Management tab in the UI.

Step 2: Create a Notification Integration

Note: The integration must be named etl_cost_monitor_email.

CREATE NOTIFICATION INTEGRATION IF NOT EXISTS etl_cost_monitor_email
  TYPE = EMAIL
  ENABLED = TRUE
  DEFAULT_RECIPIENTS = (
    'user1@example.com',
    'user2@example.com'
  );

Step 3: Grant Usage on Integration to the App

GRANT USAGE ON INTEGRATION etl_cost_monitor_email TO APPLICATION etl_cost_monitor_app;

Step 4: Create a Recurring Task

CREATE OR REPLACE TASK etl_cost_monitor_alert_task
  WAREHOUSE = <warehouse>
  SCHEDULE = '24 HOURS'
AS
  CALL etl_cost_monitor_app.code_schema.evaluate_and_alert();

Step 5: Resume the Task

ALTER TASK etl_cost_monitor_alert_task RESUME;

Note: The app cannot verify whether the task is running — please confirm manually.

Stored Procedures

etl_cost_monitor_app.code_schema.evaluate_and_alert()

Evaluates current and forecasted usage against thresholds and triggers email alerts if needed.

Application Role

etl_cost_monitor_app.app_public

Required Privileges

• USAGE on the notification integration etl_cost_monitor_email
• SELECT access on the ETL usage table is requested via the app UI when launched.

Example SQL Queries

1. Top 5 Connectors by Volume This Month

SELECT connection_name, SUM(incremental_rows) AS total_rows
FROM fivetran_metadata.incremental_mar
WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE())
GROUP BY connection_name
ORDER BY total_rows DESC
LIMIT 5;

2. Month-over-Month Growth by Connector

WITH current_month AS (
  SELECT connection_name, SUM(incremental_rows) AS current_rows
  FROM fivetran_metadata.incremental_mar
  WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE())
    AND free_type = 'PAID'
  GROUP BY connection_name
), last_month AS (
  SELECT connection_name, SUM(incremental_rows) AS last_rows
  FROM fivetran_metadata.incremental_mar
  WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE() - INTERVAL '1 MONTH')
    AND free_type = 'PAID'
  GROUP BY connection_name
)
SELECT
  c.connection_name,
  c.current_rows,
  COALESCE(l.last_rows, 0) AS last_rows,
  c.current_rows - COALESCE(l.last_rows, 0) AS growth
FROM current_month c
LEFT JOIN last_month l ON c.connection_name = l.connection_name
ORDER BY growth DESC
LIMIT 5;

3. Table-Level Usage Breakdown (Current Month)

SELECT destination_id, schema_name, table_name, SUM(incremental_rows) AS total_rows
FROM fivetran_metadata.incremental_mar
WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE())
  AND free_type = 'PAID'
GROUP BY destination_id, schema_name, table_name
ORDER BY total_rows DESC
LIMIT 10;