Edge Release Notes - November 21, 2024

Existing Sources Improvement

PostgreSQL

• Partitioning Support: Added support for table partitioning in PostgreSQL for versions 13 and above, enhancing performance and scalability when handling large datasets.

• Row size support: The platform now supports row sizes up to 100 KB. For rows exceeding this limit (up to 1 MB), please contact support.

• Improved Historical Load Performance: Enhanced data sync performance for historical data loads through various optimizations, reducing sync times and increasing reliability for large datasets.

New Sources

Platform Improvements

• 5-Minute Schedule: Introduced the option to sync data every 5 minutes now. Please contact Sales or your account executive to enable this feature for your team.

• Public API Availability: The public API allows you to manage your Pipelines programmatically, creating, updating, and deleting data Pipelines, for enhanced automation and flexibility.

• Support for Key Pair Authentication in Snowflake: Connect securely to your Snowflake Destination using key pair authentication. This method allows you to provide a private key instead of a database password while configuring your Snowflake Destination.

• Alerts: Ability to configure alert preferences, choose specific Pipelines to receive alerts for, subscribe to Pipeline alerts created by other users, and view alerts in the dashboard.

• RBAC: This feature brings advanced user management capabilities. It empowers administrators to have finer control over user permissions and access rights across various features and functionalities within the system. This ensures that users have access only to the resources necessary for their job roles, improving security and data protection.

Pipeline Reliability and Control

• Data Replication Type Configuration: Ability to configure the replication to skip historical data and replicate only incremental data to the Destination.

Pipeline Observability

• Banners for Pipeline Issues: Introduced informational banners on the Pipeline homepage to highlight any current issues, ensuring that users stay aware of potential disruptions.

• Display Skipped Jobs: When a running job extends past the next scheduled interval, you can now see the skipped jobs, improving job tracking and observability.

• Improved Job Progress Visibility: Job progress updates are now visible for jobs, providing real-time insights into job execution status for enhanced monitoring.

Upcoming Features

• New Source Integrations: Support for Salesforce, additional PostgreSQL and MySQL variants, Facebook Ads, and SQL Server CT as a Source.

• New Destination: Support for Redshift and S3 with Iceberg.

• Advanced Edits: Complete editing capabilities at the Pipeline level.

• Column-level Schema Evolution: This feature identifies newly added columns in the existing tables and replicates them along with the existing columns to the Destination.

• WAL Slot Monitoring: Users can modify the Write-Ahead Logging (WAL) slot monitoring threshold.

• Standard to Edge Pipeline Upgrades: Upgrade your Standard Pipelines to Edge with the same schema and Destination table structure. The support team will help you enable this feature.

• Parallel Historical and Incremental Ingestion: Incremental ingestion initiates while historical ingestion is still in progress for all tables in the Pipeline.

• Unquoted Identifiers in Table and Column Names: A Pipeline configuration option to manage quoted identifiers. This will allow users to enable or disable quotes around table and column names, ensuring compatibility with their existing Snowflake DDL queries.