Integration Details

Data management as a service, delivered into your infrastructure.

Databricks powers our unified timeseries lakehouse architecture, providing a scalable, AI-ready data platform that unifies financial datasets across diverse sources. You retain full ownership and control of your data in your cloud infrastructure, while we provide the comprehensive stack to integrate, normalize, govern, and deliver that data in a highly usable, joinable, and composable format.

Our integration leverages Databricks' core capabilities—Delta Lake, Unity Catalog, Delta Sharing, and the Medallion Architecture—to build a modern data platform that combines:

Unified Data Architecture:

1. Bronze Layer: Raw ingestion of PMS, SecMaster, reference data, risk models, and alternative data stored in your cloud storage (S3, ADLS)
2. Silver Layer: Cleaned and normalized into consistent timeseries format (Parquet, Delta Lake, Iceberg) with quality checks, de-duplication, and gap handling
3. Gold Layer: Enriched datasets joined via temporal resolvers to stable SecMaster IDs, ready for BI, analytics, and downstream applications

SecMaster & Cross-Dataset Resolution:

• Stable instrument IDs (FIGI, PermID) mapped to Bloomberg IDs, RIC, ISIN, SEDOL, CUSIP, and your OMS identifiers
• Point-in-time and historical resolution with manual patching workflows
• Coverage for equities, options, futures, fixed income with type-specific attributes and entity relationships

Data Sources Integration:

• Core datasets: Pricing, corporate actions, PMS, fund admin, risk models
• External datasets: Fundamental data, estimates, alternative data via Delta Sharing (S&P Global, FactSet, Bloomberg)
• Both batch and streaming ingestion for intraday use cases

Governance & Quality:

• Unity Catalog for unified governance, data lineage, and access controls
• Built-in data quality tests (DBT, Great Expectations) with escalation workflows
• AI-powered documentation and natural language search

Timeseries Analytics:

• Calendar-aware timeseries arithmetic and windowed functions
• As-of joins for point-in-time analysis
• Lags, lookbacks, and performance-optimized operations

FAQs

1. How does SecMaster ID mapping work?

We maintain stable instrument identifiers (like FIGI and PermID) that map to all your vendor identifiers—Bloomberg IDs, RIC, ISIN, SEDOL, CUSIP, and your OMS provider IDs. Our temporal resolvers perform point-in-time or historical resolution based on common instrument attributes with fallback rules. We support manual patching and override workflows with AI assistance to handle edge cases and ensure complete coverage across your portfolio and research universe.

2. What data sources do you integrate?

We integrate core datasets (pricing, corporate actions, PMS, fund admin, risk models) and external datasets (fundamental data, estimates, alternative data). For core sources, we provide drop-down menu simplicity during onboarding with pre-built resolvers and ingestion pipelines. External datasets leverage Delta Sharing to access data from providers like S&P Global, FactSet, and Bloomberg directly in your lakehouse, plus custom alternative data sources through standard pipelines with coverage and quality monitoring.

3. How is data governance handled?

We implement governance using Databricks Unity Catalog, providing unified data and AI asset management with automated column-level lineage, row and column-level access controls, and sensitive data classification (e.g., PII). Data quality is enforced through DBT tests and Great Expectations with escalation workflows. All transformations are version-controlled with CI/CD pipelines, and AI-powered documentation provides natural language search and auto-generated metadata across your data assets.

4. What about data ownership and deployment?

You retain complete ownership and control of your data. The entire stack is deployed into your cloud infrastructure (AWS, Azure, GCP) or hybrid model, respecting your privacy and compliance requirements. The data flows directly between providers and your environment—we provide the engineering and platform to make it seamless, but you own the data and infrastructure.

5. Can you handle intraday and streaming data?

Yes, our platform supports both batch and streaming ingestion, enabling intraday use cases. We can process real-time market data, trade executions, and portfolio updates throughout the trading day, maintaining the same quality checks and governance standards across both batch and streaming pipelines.

6. What if my firm doesn't have a Databricks account?

If you currently don't have a Databricks account but would like to scale your data platform with it, our team can work with you to set it up, connect it to your cloud subscription, and migrate your data to it.