Centralizing data pipeline management with Matillion Hub involves
leveraging the Data Productivity Cloud to unify ETL/ELT tasks, visibility, and security in a single, browser-based interface
The Hub acts as a central control plane, connecting to various data sources and cloud data platforms (Snowflake, Databricks, Redshift) while using agents for execution.
Here are the key requirements, components, and steps to centralize management:
Centralized Infrastructure (Control Plane)
• Deployment Options: Choose between full SaaS (Matillion hosts and manages infrastructure) or hybrid-SaaS (you deploy and manage agents in your private cloud).
• Agent Management: Use the Hub to manage, monitor, and configure data plane agents that run your pipelines.
• Git Integration: Enable Git features within the Hub to manage version control, allowing teams to contribute, push transformations, and manage data products centrally.
Data Source and Destination Integration
• Connectivity: Configure pre-built or custom connectors in the Hub to move data from databases (PostgreSQL), SaaS applications (, Google Analytics), or files (Amazon S3).
• Target Configuration: Define the default cloud data warehouse (Snowflake, etc.) for ELT processes.
Operational Requirements (Management)
• Shared Pipelines: Create reusable, shared pipelines to avoid duplicating logic across projects, ensuring consistency in auditing, data masking, and transformations.
• Advanced Scheduling: Use the Scheduler in the Designer interface to manage, monitor, and fine-tune pipeline execution times using either standard or cron expressions.
• Visibility & Monitoring: Use the centralized Dashboard to monitor all pipeline activities, view logs, and troubleshoot with a single click.
• User Roles & Security: Assign user permissions and roles within the Hub to ensure governance.
Development & Transformation
• Low-Code/High-Code Interface: Utilize the Designer interface within the Hub for visual, low-code transformation, or incorporate Python/SQL scripting.
• DataOps Enabled: Use the Matillion Pipeline Language (DPL) to automate the creation of folders, projects, and pipelines, integrating with tools like Azure DevOps.