Fusion Metadata Registry

SDMX Structural Metadata Registry

Fusion Metadata Registry is a free to use structural metadata registry for SDMX supporting EDI, SDMX 1.0, 2.0, 2.1 and 3.0.

Structural Metadata Registry.
Author and maintain SDMX structures using the web user interface or programmatically through the REST API.
Data validation - validates that data files are correctly structured.
SDMX structure mapping - transform SDMX data to a different DSD and coding.
SDMX data conversion - convert data files between SDMX data formats and versions.
Download and install the Java web application, or deploy as a container.

Sponsored by the Bank for International Settlements

License

Web application: Public License Agreement
Source code: Apache 2.0 License Agreement
License Agreement

Download Docker

BIS Open Tech

Fusion Metadata Registry is published under the BIS Open Tech initiative.

The BIS Open Tech is a platform for sharing statistical and financial software as public goods, by promoting international cooperation and coordination. These software tools are developed according to international best practices and standards and can be reused and further developed in a wide variety of environments.

FMR helps official statistics organisations externalise and gain control of their statistical structural metadata with the benefits of maintainability, re-use, standardisation, harmonisation and improved metadata and data governance.

FMR provides:

A controlled repository for SDMX structural metadata, which acts as a single source of truth for: automated metadata-driven data processing systems, metadata needed by data providers in data collections, structures for the purposes of standardisation, statistical data harmonisation, and communal benefit. Implements the SDMX 3.0 Registry specification.
Three integrated metadata-driven data processing services: data validation, data transformation (mapping data between structures) and conversion of data between SDMX data formats. Usable through the web UI or integrated into data automation processes using the REST API.
Metadata authoring and maintenance web UI, enabling metadata managers to browse, author and maintain their structural metadata model.

Fusion Metadata Registry 11 released

Fusion Metadata Registry version 11 implements the SDMX 3.0 specification.

Download get the Docker image, or read the release notes

Backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes.
Note that FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.

introductory video

Inside Fusion Metadata Registry

The heart of Fusion Metadata Registry is its structural metadata repository - essentially a database that implements the SDMX 3.0 information model.

All structural metadata objects are stored according to this model which provides a rigorous dependency graph that defines and enforces the metadata's referential integrity. This strongly enforced referential integrity provides confidence that the metadata model is always complete - a DSD for instance can never exist without its referenced Codelists.

FMR provides two interfaces

Web user interface: a web interface for interactively discovering and retrieving structures, structure authoring and maintenance, and system administration.
REST application programming interface: a REST web services API for automation that implements the SDMX v2 structure REST API specification, with additional extensions for creating and modifying structures, data validatio and data transformation.
- FMR 10 implements the SDMX 2.1 registry specification.
- FMR 11 implements the SDMX 3.0 registry specification.

In addition to its core role as a structural metadata registry, FMR's integrated data processing services support:

Data validation: submitted data is checked against nine separate quality rules including consistency with the DSD, and compliance with any Data Constraints
Data conversion: conversion of SDMX datasets between transmission formats including EDI, XML, JSON, CSV and Excel
Data mapping: transformation of a dataset to a different DSD using SDMX structure mapping

Finally for for data collectors, FMR's Excel data collection forms generator generates cross-tabulation data collection forms allowing data reporters to simply fill in the blank cells with the required observation values. The precise content and layout of these forms are defined by creating templates from which a personalised form is generated for each data provider.

Key Use Cases

Structural metadata management

Centralise and control structural metadata
Author and maintain structural metadata
Drive statistics production using centrally-controlled metadata
Provide an SDMX-compliant a REST structure API

Statistics production

Validate data for compliance with structural metadata, constraints and business rules (not VTL)
Execute complex data mapping (SDMX structure maps, not VTL)
Convert data between formats (convert between SDMX formats)

Data collection

Collect data using metadata-driven Excel forms
Map incoming data into standardised structures

Data reporting

Prepare SDMX data for reporting
Validate data for compliance with the data collector's specifications prior to reporting

Capacity

No fixed limits on the number of users
No fixed limits on the volume of structures - in practice the volume of structures is limited by the available compute platform memory

Features

Feature list

Feature	FMR 10	FMR 11
Internal Information Model
SDMX 2.1
SDMX 3.0
Structural Metadata Formats
SDMX-ML v3.0
SDMX-ML v2.1
SDMX-ML v2.0
SDMX-ML v1.0
SDMX-JSON v2.0
SDMX-JSON v1.0
EDI
Structural Metadata Management
Load structural metadata from file
Load structural metadata from URL
Author and maintain structural metadata using the web interface
Author and maintain structural metadata using FusionXL addin for Excel
ACID structure repository
Structural metadata import / export as SDMX
Structural metadata import / export as Excel
SDMX structures REST API
Structure storage (limited by available memory)
Structure comparison
Structure validation on import
Structure referential integrity enforcement
Tools to correct referential integrity issues (fix 'zombie' structures)
Time dependant constraints
Structure-level validity
Item-level validity
Subscription / notification of changes to structures via email or HTTP POST
Structure update transaction journalling
Rollback structure changes to a specified journalled transaction
Structural metadata time travel
SDMX Structures
Agency Schemes
Constraints
Categorisations
Category Schemes
Codelists
Concept Schemes
Data Consumer Schemes
Data Provider Schemes
Data Structure Definitions
Dataflows
Hierarchical Codelists
Metadata Structure Definitions
Metadataflows
Organisation Unit Schemes
Processes
Provision Agreements
Reporting Taxonomies
Structure Sets
Structure Map
Representation Map
Organisation Scheme Map
Concept Scheme Map (SDMX 3.0)
Category Scheme Map
Reporting Taxonomy Map
Value List
Hierarchy
Hierarchy Association
Metadata Constraint
Data Constraint
Metadata Constraint
Metadata Provision Agreement
Metadata Provider Scheme
Metadataset
'Extended' Structures (structures that are not part of the formal SDMX specification)
Validation Schemes
Reporting Templates
Publication Tables
Structural Metadata Storage Options
MySQL
Oracle
Microsoft SQL Server
Identity and Access Management
Microsoft Active Directory service integration
OpenLDAP directory service integration
Fusion Security directory service integration
Single-user mode (no directory service required)
Certificate authentication
SSO
Data
Load data from file
Load data from URL (file or REST API)
SDMX-ML v3.0 structure specific data format (XML)
SDMX-ML v2.1 structure specific data format (XML)
SDMX-ML v2.0 structure specific data format (XML)
SDMX-ML v2.1 generic data format (XML)
SDMX-ML v2.0 generic data format (XML)
SDMX-JSON v1.0 data format
SDMX-JSON v2.0 data format
SDMX-CSV v1.0 data format
SDMX-CSV v2.0 data format
EDI data format
Excel FusionXL data format
Interactive data load using the web user interface
Data storage
Data processing REST API (validation, conversion, mapping)
Data storage REST API
Plugin interface for adding custom data readers
Data Registration
SDMX data source registration
Validation and indexing of registered data sources
Reference Metadata
Store Reference Metadata Reports
SDMX-ML v2.1 reference metadata report format read
SDMX-JSON v2.0 reference metadata report format
Reference metadata submission API (POST)
Reference metadata retrieval API (GET)
Reference metadata deletion API (DELETE)
Data Validation
Structural compliance
Compliance of data values with the DSD
Compliance with constraint rules
Duplicate observations or series checks
Mandatory attributes
Validation Schemes - basic business rule validation expressions
Validation Schemes - advanced business rule validation expressions
Validation rules configuration - set what rules are applied and whether they prevent conversion or data publication
Plugin interface for adding custom validators
Data validation processing (load data, validate it against the defined structure and business rules, return a validation report plus valid and invalid series as separate output datasets)
Data validation REST API
SDMX-ML v2.1 Structure Specific data XML schema validation
SDMX-ML v3.0 Structure Specific data XML schema
VTL 2.0
Storage of VTL programs as SDMX structural metadata
Execution of VTL programs for validation or transformation of data
SDMX Structure Mapping
Concept Scheme Maps (SDMX 2.1)
Codelist Maps
Data Structure Maps
Dataflow Maps
Regex pattern mapping rules
N-to-N data maps
N-to-1 data maps
1-to-N data maps
1-to-1 data maps
Mapped Datasets
Mapped Datasets - like SQL views
Virtual Dataflows
Virtual Dataflows describe a sub-cube of a dataflow
Data Transformation
Data transformation processing (load data, transform it to a different structure using SDMX Structure Mapping and return the result)
Data transformation REST API
Data Conversion
Between supported SDMX formats and versions (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI)
From Excel Reporting Template to SDMX formats
Quick convert to SDMX-ML 2.1 Structure Specific format
Conversion to and from Fusion Excel format for SDMX datasets
Data Reporting
Excel Reporting Templates - definition and generation
Excel Reporting Templates - validation and conversion of reported data
Excel Reporting Templates - load and storage of reported data
Excel Reporting Templates - configurable template security rules
Content Security / Access Controls
Structure creation and maintenance controls (Agency privileges)
Data load and modification controls (Data Provider privileges)
Data access controls at the dataflow, series or observation level
Structure access controls at the structure or item level
Public mode - Registry content is readable by both unauthenticated guest and authenticated users
Private mode - Registry content is accessible only to authenticated users
Auditing
SDMX structure transaction auditing
User activity auditing
Enterprise Deployment
Load balancer support (clustering multiple Registry instances behind a load balancer for resilience and load tolerance)
Apache Kafka Integration
Kafka 'Structure Notification' Producer for publishing changes to structural metadata on a definable topic
Client-side API
IM-JS JavaScript client-side programming library
SDMX-UI TypeScript client-side programming library (higher-level abstraction)
Command Line Interface
Fusion Command Line Interface for managing structural metadata

2023 Roadmap

Prioritised General Enhancements workstream

Complete

Publication tables

FMR 11.1.4

New presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)

Author, validate, map and transform data using Excel

FMR 11.1.5

Load data from Excel workbooks in FusionXL data format
Use the FusionXL Excel add in for FMR to author, validate and transform SDMX datasets

Improved data processing performance

FMR 11.1.6

Improved performance and reduced memory requirements when loading, validating, mapping and converting large datasets

Reference metadata

FMR 11.2.0

Author Metadata Structure Definitions and Metadataflows
Store and retrieve reference metadata reports (metadatasets)
more...

Certificate authentication

FMR 11.2.0

Use X509 certificates to automatically authenticate with FMR's secure web services without the need to pass username / password credentials

SDMX Data Registration

Support for registering SDMX data sources

On work plan

Metadata time travel: Browse the structural metadata registry at a point of time, or query for structures as they were on a particular date
SDMX-JSON 3.0 structure format: Support for the SDMX 3.0 JSON transmission format for structural metadata
Improved error handling framework: Improved capture, handling and reporting of errors such as referential integrity violations
Structural metadata consistency checks and 'safe mode': Strengthened checks on Registry start-up with the Registry placed in 'safe mode' if problems are detected
Performance improvements: Improved support for very large structures, improved performance when processing large datasets
Metadata time travel: Retrieve structual metadata as it was at a defined point in the past
Structure mapping default observation value: Option to to set a default ouput observation value for use where the input value is missing
SDMX-JSON 3.0 data format: Support for the SDMX 3.0 JSON transmission format for data excluding hierarchical reference metadata embedded within the data message
Virtual dataflows: Virtual dataflows describe a sub-cube of data similar in concept to a SQL view

Data processing microservices workstream

On work plan

Mapping microservice: Decoupled stateless SDMX mapping microservice

Recent features

See the full changelog for details of all new features and bug fixes.

11.6.0 - March 2023

Data Registration support
Publication Tables - support a scaling function applied to numerical observations
Transformation performance - option to skip the validation phase to improve performance and reduce memory pressure
Large codelists performance when using item validity

11.5.0 - December 2022

Mapping - Improve Metrics Report with IncUnmappedReport option
Publication Tables - support numerical observation formatting.
Publication Tables - support set scaling factor.
Publication Tables - support categorisation of publication tables via the FMR UI.
Support for "references-partial" in regular expressions.
Add auditing for structure queries.

11.4.0 - October 2022

Dependencies incompatible with the Apache 2.0 open source license removed.
Publication Tables - support casting of dates more than once in the same expression.

11.3.0 - September 2022

Migration of structure change history when upgrading from FMR 10 to 11, cloning FMR environments or for backup / recovery purposes.

11.2.0 - July 2022

Reference metadata support
Certificate authentication support

11.1.7 - July 2022

Interactive UI for synchronising structures between FMR environments such as TEST and LIVE - compare the contents of two FMR environments and push / pull selected differences
Publication Tables footnotes support

11.1.6 - June 2022

New REST API web service parameter to control how duplicate obervations are handled on validation and transformation
Introduction of MVStore for temporary storage and consolidation of datasets during processing operations delivering improved performance and reduced memory demand

11.1.5 - June 2022

MySQL 8 support
Publication Tables: Option to output rows with no data
Publication Tables: Dimension Value based on another Dimenisons variable value
Publication Tables: CPMI - Dynamic Rows and columns
Load data from Excel workbooks in Fusion Excel data format

11.1.4 - May 2022

Publication tables - new presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)

11.1.3 - May 2022

maintenance release

11.1.2 - April 2022

maintenance release

11.1.1 - February 2022

maintenance release

11.1.0 - February 2022

Code refactored to use the new SDMX 3.0 io.sdmx Java packages
New validation rule to check that OBS_PRE_BREAK is consistent with OBS_STATUS

11.0.2 and 11.0.3 - February 2022

maintenance releases

11.0.1 - January 2022

SDMX 2.1 categorisations against hierarchical code lists applied to all relevant SDMX 3.0 hierarchies

11.0.0 - December 2021

Internal information model changed to SDMX 3.0 (FMR 10 uses the SDMX 2.1 information model internally)
Support for storage of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map
Support for SDMX 3.0 XML format for structures
Support for SDMX 3.0 CSV format for data
Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load
Semantic versioning - support for three-digit version numbers
JNDI database connection support

More...

10.7 - May 2021

Option to send structures as stubs on Kafka
Data validation - new option to supply the SDMX Sender ID
Mappings can be applied to multiple datasets if they refer to the same dataset
Support the creation of the TIME_SERIES attributes from an EDI file

10.6

Improved Apache Kafka integration
Fusion Metadata Registry acts as an Apache Kafka producer publishing changes to the SDMX structures on definable topics. This is useful in metadata-driven statistics production architectures where distributed applications like data validators need to hold a synchronised copy of the master structural metadata.
In 10.6, the Kafka interface has been enhanced to support all connection properties, email alerts are sent if the producer terminates, filters have been added to select the structures to publish and publication failure recovery has been strengthened. more

10.5

Transaction and user activity auditing
Auditing of changes to SDMX structural metadata artefacts and user activity more
Structure transaction rollback
Journalling of changes to the structural metadata content allowing administrators to rollback to a specific point in time more
Server integrity
Checks the referential integrity of the structural metadata allowing administrators to fix problems found like 'zombie' artefacts that have broken references more
Docker image for containerised deployment
Fusion Metadata Registry is now available as an image on Docker Hub for faster containerised deployment with minimal setup quick start