Forem: Anuar Ustayev

Talk to your data from any chat app

Anuar Ustayev — Tue, 17 Feb 2026 11:43:00 +0000

Most data tools assume you’ll go to them.

You log into a dashboard.
You click through filters.
You hope someone built the chart you need last quarter.

But that’s not how people actually work.

When you need a quick number, you don’t open BI software. You send a message:

“What were our numbers last month?”

The answer comes back in the same thread.

So the obvious question is: why can’t data work the same way?

Meet data where work already happens

Instead of forcing users into another interface, Queryless runs inside chat apps — Telegram today, others soon.

You ask a question in plain English.
It runs the query in real time.
It returns a clean, structured answer — often with a markdown table and, when useful, a link to an interactive report.

No dashboards.
No new tabs.
No context switching.

Just a message.

Why chat is the right abstraction

Chat has a few properties that traditional data tools don’t:

It’s already where collaboration happens
It’s asynchronous and searchable
It’s mobile-first by default
It lowers the barrier to asking “simple” questions

For many use cases, people don’t need a complex dashboard. They need a number, a ranking, or a comparison — fast.

Example:

You open the bot in Telegram and type:

“What are the top 5 countries by GDP in 2023?”

The response comes back with:

A short summary
A properly formatted table
Exact figures
A link to an interactive report (for deeper exploration)

All within seconds. No UI hunting required.

Who this is actually for

This isn’t about replacing BI platforms.

It’s about reducing friction for:

Executives checking numbers between meetings
Analysts who need quick comparisons without opening spreadsheets
Team leads sharing verified data in group chats
Developers who want a lightweight way to query structured datasets

If someone has a question about data, they shouldn’t need to become a data analyst to answer it.

The bigger shift

The interesting part isn’t just “chatbots for data.”

It’s the interface shift.

We’ve trained ourselves to believe data must live in dashboards. But dashboards are optimized for exploration — not for answering single questions quickly.

Chat flips the model:

Question first
Structured answer immediately
Optional deeper dive

As LLMs improve and structured querying becomes more reliable, embedding data access directly into communication channels feels less like a feature — and more like the default.

Try it

There’s a public demo agent running on Telegram with economic and commodity datasets (GDP, inflation, oil, gold, natural gas, etc.).

Open the bot and ask anything.

The real test isn’t how flashy it looks.

It’s whether you ever feel like opening a dashboard again.

The Metadata Standards Landscape: Making Data Discoverable Across Organizations

Anuar Ustayev — Tue, 08 Jul 2025 00:00:00 +0000

Introduction

In our previous exploration of metadata and data discovery, we saw how rich metadata transforms search and discovery in data catalogs. But here's the challenge: rich metadata without standards creates chaos. When every organization describes data differently, you get the digital equivalent of Tower of Babel—lots of information, zero interoperability.

Consider this scenario: A data portal contains datasets from various departments within a government agency. Without metadata standards, the transportation department describes their datasets using fields like "author," "created_date," and "topic." The health department uses "publisher," "last_modified," and "subject_area." The education department uses "creator," "updated," and "category." All contain the same core information, but without standardized metadata fields, users can't consistently search, filter, or understand what's available across the portal.

This is where metadata standards come in—providing the consistent structure that makes data discoverable and manageable within your portal, while also enabling interoperability across the broader data ecosystem.

The Major Players: Core Metadata Standards

The metadata standards landscape might seem overwhelming at first, but it's built around a few foundational standards that work together rather than compete. Let's explore the key players and how they complement each other.

Dublin Core: The Universal Foundation

Dublin Core is the veteran of metadata standards with its 15 basic elements that can describe virtually any resource. Published as ISO Standard 15836, it's domain-agnostic and internationally recognized.

The 15 Core Elements:

# Dublin Core basics
title: "NYC 311 Service Requests - 2023 Q1"
creator: "NYC Department of Information Technology"
subject: "citizen services, municipal data, complaints"
description: "Citizen complaints and service requests from NYC's 311 system"
publisher: "NYC Open Data"
contributor: "NYC 311 Contact Center"
date: "2023-04-01"
type: "Dataset"
format: "CSV"
identifier: "https://data.cityofnewyork.us/311-2023-q1"
source: "NYC 311 System"
language: "en"
relation: "Part of NYC Open Data Portal"
coverage: "New York City, 2023 Q1"
rights: "Public Domain"

Why Dublin Core Works:

Universal applicability - Works for datasets, documents, images, anything
Low barrier to entry - Easy to understand and implement
Foundation for other standards - DCAT, MODS, and others build on Dublin Core
Proven longevity - 25+ years of development and refinement

DCAT: Purpose-Built for Data Catalogs

Data Catalog Vocabulary (DCAT) is Dublin Core's specialized cousin, designed specifically for describing datasets in data catalogs. As a W3C Recommendation, it's the gold standard for data portal interoperability.

DCAT's Key Additions:

# DCAT builds on Dublin Core with data-specific concepts
dataset:
  title: "NYC 311 Service Requests - 2023 Q1"
  description: "Citizen complaints and service requests..."
  publisher: "NYC Department of Information Technology"
  theme: "Government, Public Services"
  keyword: ["311", "citizen services", "complaints"]
  landingPage: "https://data.cityofnewyork.us/311-2023-q1"

  # Data-specific metadata Dublin Core lacks
  distribution:
    - downloadURL: "https://data.cityofnewyork.us/311-2023-q1.csv"
      mediaType: "text/csv"
      format: "CSV"
    - downloadURL: "https://data.cityofnewyork.us/311-2023-q1.json"
      mediaType: "application/json"
      format: "JSON"

  temporal: "2023-01-01/2023-03-31"
  spatial: "New York City, NY, USA"
  accrualPeriodicity: "quarterly"
  contactPoint:
    fn: "NYC Open Data Team"
    hasEmail: "opendata@cityhall.nyc.gov"

DCAT's Power:

Federated search - Multiple catalogs can be searched as one
Automated harvesting - Systems can automatically collect metadata from DCAT-compliant catalogs
Distribution management - Tracks multiple formats (CSV, JSON, API) of the same dataset
Data catalog-specific concepts - Update frequency, spatial/temporal coverage, data quality

Schema.org Dataset: Web-Native Discovery

Schema.org Dataset brings metadata into the web era, designed specifically for search engine optimization and web-based discovery.

Schema.org in Practice:

<!-- Embedded in webpage HTML for search engines -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Dataset",
  "name": "NYC 311 Service Requests - 2023 Q1",
  "description": "Citizen complaints and service requests from NYC's 311 system during January-March 2023",
  "url": "https://data.cityofnewyork.us/311-2023-q1",
  "creator": {
    "@type": "Organization",
    "name": "NYC Department of Information Technology"
  },
  "temporalCoverage": "2023-01-01/2023-03-31",
  "spatialCoverage": "New York City, NY, USA",
  "distribution": [
    {
      "@type": "DataDownload",
      "encodingFormat": "CSV",
      "contentUrl": "https://data.cityofnewyork.us/311-2023-q1.csv"
    }
  ]
}
</script>

Schema.org's Advantage:

Google Dataset Search - Direct integration with Google's dataset discovery platform – https://datasetsearch.research.google.com/
SEO benefits - Search engines understand and rank your data
Web-standard JSON-LD - Fits naturally into web development workflows
Rich snippets - Enhanced search results with download links and previews

Standards in Practice: Who Uses What

Different sectors have gravitated toward different standards based on their specific needs, organizational culture, and regulatory requirements.

Government Sector: Transparency and Interoperability

Government data portals prioritize standardization and cross-agency interoperability, leading to strong adoption of DCAT-based approaches.

United States Federal Government:

The Project Open Data Metadata Schema, mandated for federal agencies, builds directly on DCAT:

# Project Open Data Schema (DCAT-based)
title: "Federal Student Aid Recipients by State"
description: "Annual data on federal student aid recipients and amounts by state and program type"
keyword: ["education", "financial aid", "students", "federal programs"]
modified: "2023-09-15"
publisher: "Department of Education"
contactPoint:
  fn: "Federal Student Aid Data Team"
  hasEmail: "data@ed.gov"
mbox: "data@ed.gov"
identifier: "ED-FSA-2023-001"
accessLevel: "public"
bureauCode: ["018:00"]
programCode: ["018:001"]
license: "https://creativecommons.org/publicdomain/zero/1.0/"
spatial: "United States"
temporal: "2022-07-01/2023-06-30"

European Union:

DCAT-AP (DCAT Application Profile) extends DCAT with European-specific requirements:

# DCAT-AP adds EU-specific fields
dataset:
  # Standard DCAT fields
  title: "EU Carbon Emissions by Sector"
  description: "Annual greenhouse gas emissions data by economic sector across EU member states"

  # DCAT-AP extensions
  conformsTo: "https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32019R2152"
  page: "https://ec.europa.eu/eurostat/carbon-data"
  versionInfo: "2.1"
  hasVersion: "https://data.europa.eu/carbon-emissions-v2.1"
  language: ["en", "fr", "de"]
  provenance: "Collected from national statistical offices via standardized reporting"

Benefits for Government:

Cross-agency compatibility - Datasets from different agencies can be easily combined
Automated compliance checking - Tools can validate metadata completeness
Public transparency - Citizens can find and understand government data
International data sharing - Compatible metadata enables collaboration between countries

Academic and Research: Preservation and Scholarly Discovery

Academic institutions balance standardization with discipline-specific needs, often layering specialized standards on Dublin Core foundations.

Research Data Repositories:

# Dublin Core + research-specific extensions
title: "Longitudinal Study of Urban Heat Islands - Chicago 2015-2023"
creator: ["Dr. Sarah Chen", "Dr. Michael Rodriguez", "Climate Research Lab"]
subject: "climatology, urban planning, environmental science"
description: "Temperature measurements from 150 sensors across Chicago measuring urban heat island effects over 8 years"
publisher: "University Research Data Repository"
date: "2023-08-15"
type: "Dataset"
format: ["CSV", "NetCDF"]

# Research-specific additions
fundingAgency: "National Science Foundation"
grantNumber: "NSF-GEO-1847392"
methodology: "Fixed sensor networks with 15-minute measurement intervals"
relatedPublication: "doi:10.1038/climate.2023.456"
ethicsApproval: "IRB-2015-378"
dataCollectionPeriod: "2015-06-01/2023-05-31"
spatialResolution: "100m grid"
temporalResolution: "15 minutes"

Social Science Data (DDI Standard):
The Data Documentation Initiative provides rich metadata for survey and social science data:

# DDI (Data Documentation Initiative) for social science
study:
  citation:
    title: "American Community Survey 2023"
    creator: "U.S. Census Bureau"
    producer: "U.S. Census Bureau"
    distributionDate: "2024-03-15"

  dataCollection:
    methodology: "Probability sample survey"
    samplingProcedure: "Stratified systematic sampling"
    collectionMode: "Mail, telephone, and in-person interviews"
    responseRate: 97.2

  variableGroups:
    demographics:
      variables: ["age", "gender", "race", "ethnicity"]
    economic:
      variables: ["income", "employment_status", "occupation"]
    housing:
      variables: ["housing_type", "tenure", "housing_costs"]

Academic Priorities:

Long-term preservation - Metadata must support data archiving for decades
Citation tracking - Clear links between datasets and publications
Methodology documentation - Detailed information about data collection methods
Interdisciplinary discovery - Subject classifications that work across fields

Nonprofits: Balancing Standards and Resources

Nonprofit organizations often follow government standards for compatibility while adapting to resource constraints and mission-specific needs.

# Nonprofit metadata - simplified DCAT approach
title: "Global Health Initiative: Vaccination Rates 2020-2023"
description: "Country-level vaccination coverage data collected through partnerships with WHO and national health ministries"
publisher: "Global Health Alliance"
theme: "Health, International Development"
keyword: ["vaccination", "global health", "immunization", "public health"]
license: "CC-BY-4.0"
contactPoint:
  fn: "Data Team"
  hasEmail: "data@globalhealthalliance.org"

# Mission-specific fields
programArea: "Vaccine Equity Initiative"
partnerOrganizations: ["WHO", "UNICEF", "Gavi Alliance"]
impactMetrics:
  - "Lives saved through improved vaccination access"
  - "Healthcare systems strengthened"
fundingSource: "Private foundation grants"
dataUseAgreement: "Must acknowledge source and share derivative works"

Enterprise vs. Public Portals: Different Needs, Different Standards

The choice of metadata standards often depends on whether your data portal serves internal enterprise needs or public discovery. Each context brings distinct requirements and constraints.

Public Data Portals: Standardization First

Public-facing data portals prioritize interoperability and discoverability, leading to strict adherence to established standards.

Characteristics of Public Portal Metadata:

Strict DCAT compliance - Enables cross-portal search and data harvesting
Schema.org integration - Ensures Google and other search engines can index datasets
Quality assurance at scale - Automated validation of metadata completeness and accuracy
Multi-language support - Metadata in multiple languages for international accessibility

# Public portal: strict standards compliance
dataset:
  # Required DCAT fields - enforced by validation
  title: "Public Transit Routes and Schedules"
  description: "Real-time and scheduled route information for city public transportation"
  publisher: "Metropolitan Transit Authority"
  issued: "2023-01-15"
  modified: "2023-06-29"

  # Standardized vocabularies
  theme: "http://publications.europa.eu/resource/authority/data-theme/TRAN"
  keyword: ["transportation", "public transit", "GTFS", "real-time"]

  # Interoperability requirements
  conformsTo: "https://gtfs.org/reference/static"
  accessRights: "http://publications.europa.eu/resource/authority/access-right/PUBLIC"
  license: "https://creativecommons.org/licenses/by/4.0/"

  # Multiple format support
  distribution:
    - downloadURL: "https://transit.city.gov/gtfs/routes.zip"
      mediaType: "application/zip"
      format: "GTFS"
    - accessURL: "https://api.transit.city.gov/routes"
      mediaType: "application/json"
      format: "JSON API"

Internal Enterprise Portals: Flexibility and Integration

Enterprise data portals start with standards but customize heavily for business needs, integration requirements, and organizational workflows.

Enterprise Portal Adaptations:

# Enterprise portal: standards + business context
dataset:
  # Standard fields (DCAT-based)
  title: "Customer Purchase History - Q2 2023"
  description: "Anonymized customer transaction data for business intelligence and analytics"
  publisher: "Data Engineering Team"

  # Business-specific extensions
  businessDomain: "Customer Analytics"
  dataOwner: "VP Customer Experience"
  technicalContact: "data-engineering@company.com"
  businessContact: "customer-analytics@company.com"

  # Governance and compliance
  dataClassification: "Internal - Confidential"
  retentionPeriod: "7 years"
  complianceFramework: ["GDPR", "CCPA", "SOX"]
  approvalStatus: "Approved for Business Use"
  lastAuditDate: "2023-06-15"

  # Technical integration
  sourceSystem: "SAP Customer Management"
  refreshFrequency: "Daily at 2:00 AM UTC"
  dataLineage: "Customer DB -> ETL Pipeline -> Data Warehouse -> Analytics View"
  qualityScore: 94.2
  slaTarget: "99.5% availability, <4 hour data latency"

  # Access control (enterprise-specific)
  accessLevel: "Department Level"
  approvedRoles: ["Customer Analytics Team", "Marketing Analysts", "Product Managers"]
  dataUseAgreement: "Internal use only, no external sharing without Data Office approval"

Key Differences in Practice

1. Governance Requirements

Public Portals:

Transparency and open access focus
Standardized licenses (CC, Public Domain)
Public accountability for data quality

Enterprise Portals:

Complex access controls and permissions
Business approval workflows
Integration with identity management systems

2. Discovery Mechanisms

Public Portals:

SEO optimization for web search engines
Cross-portal federation and harvesting
Social media and link sharing

Enterprise Portals:

Integration with business intelligence tools
Internal search and recommendation systems
Context-aware suggestions based on user role

3. Quality and Validation

Public Portals:

Automated quality checks against standards
Public feedback and error reporting
Reputation and trust metrics

Enterprise Portals:

Business rule validation
Integration with data quality tools
SLA monitoring and alerting

Frictionless Data: Datopian's Approach to Simplicity

While standards like DCAT and Dublin Core provide excellent frameworks, implementing them in real-world systems often becomes complex. This is where Frictionless Data shines—offering a pragmatic approach that bridges the gap between standards and practical implementation.

The Philosophy: Progressive and Practical

Frictionless Data, developed through a joint stewardship between the Open Knowledge Foundation and Datopian, embodies a "progressive, incrementally adoptable" philosophy. Rather than forcing organizations to adopt complex standards all at once, it provides a pathway to better metadata that grows with your needs.

Core Design Principles:

Simplicity over complexity - Start simple, add sophistication gradually
Developer-friendly - Easy to implement and maintain
Standards-compatible - Works alongside and bridges to other standards
Real-world tested - 10+ years of iteration with open data communities

Core Specifications: Building Blocks for Any System

Frictionless Data provides four core specifications that work together or independently:

1. Data Package: The Container

{
  "name": "customer-transactions-q2-2023",
  "title": "Customer Transactions Q2 2023",
  "description": "Anonymized customer purchase data for Q2 business analysis",
  "version": "1.2.0",
  "licenses": [
    {
      "name": "CC-BY-4.0",
      "title": "Creative Commons Attribution 4.0",
      "path": "https://creativecommons.org/licenses/by/4.0/"
    }
  ],
  "resources": [
    {
      "name": "transactions",
      "path": "transactions.csv",
      "title": "Customer Transaction Records",
      "description": "Individual transaction records with product and customer details",
      "schema": "schema/transactions-schema.json"
    }
  ],
  "contributors": [
    {
      "title": "Data Engineering Team",
      "email": "data@company.com",
      "role": "author"
    }
  ],
  "created": "2023-07-01"
}

2. Table Schema: Structure Definition

{
  "fields": [
    {
      "name": "transaction_id",
      "type": "string",
      "title": "Transaction ID",
      "description": "Unique identifier for each transaction",
      "constraints": {
        "required": true,
        "unique": true
      }
    },
    {
      "name": "purchase_date",
      "type": "date",
      "title": "Purchase Date",
      "description": "Date when the transaction occurred",
      "format": "%Y-%m-%d"
    },
    {
      "name": "amount",
      "type": "number",
      "title": "Transaction Amount",
      "description": "Total transaction amount in USD",
      "constraints": {
        "minimum": 0
      }
    },
    {
      "name": "product_category",
      "type": "string",
      "title": "Product Category",
      "description": "High-level product category",
      "constraints": {
        "enum": ["Electronics", "Clothing", "Home", "Books", "Sports"]
      }
    }
  ]
}

3. Data Resource: Individual Asset Metadata

{
  "name": "sales-summary",
  "path": "https://data.company.com/sales/summary.json",
  "title": "Monthly Sales Summary",
  "description": "Aggregated sales data by product category and region",
  "format": "json",
  "mediatype": "application/json",
  "encoding": "utf-8",
  "schema": {
    "fields": [
      {
        "name": "month",
        "type": "date",
        "format": "%Y-%m"
      },
      {
        "name": "region",
        "type": "string"
      },
      {
        "name": "total_sales",
        "type": "number"
      }
    ]
  }
}

Why Frictionless Data Matters for Implementation

1. Standards Bridge
Frictionless Data doesn't compete with DCAT or Dublin Core—it complements them. You can easily map Frictionless metadata to other standards:

# Python example: Frictionless to DCAT mapping
from frictionless import Package

# Load Frictionless Data Package
package = Package('datapackage.json')

# Map to DCAT
dcat_metadata = {
    'dcat:Dataset': {
        'dct:title': package.title,
        'dct:description': package.description,
        'dcat:keyword': package.keywords,
        'dct:issued': package.created,
        'dcat:distribution': [
            {
                'dcat:downloadURL': resource.path,
                'dct:format': resource.format,
                'dcat:mediaType': resource.mediatype
            }
            for resource in package.resources
        ]
    }
}

2. Incremental Adoption
Start with basic Frictionless Data Package, then add sophistication:

// Week 1: Basic package
{
  "name": "sales-data",
  "resources": [
    {"name": "sales", "path": "sales.csv"}
  ]
}

// Month 1: Add schema validation
{
  "name": "sales-data",
  "resources": [
    {
      "name": "sales",
      "path": "sales.csv",
      "schema": "sales-schema.json"
    }
  ]
}

// Month 3: Full metadata
{
  "name": "sales-data",
  "title": "Monthly Sales Data",
  "description": "Comprehensive sales analytics dataset",
  "contributors": [...],
  "licenses": [...],
  "resources": [
    {
      "name": "sales",
      "path": "sales.csv",
      "schema": "sales-schema.json",
      "title": "Sales Transactions",
      "description": "Individual sales records with customer and product details"
    }
  ]
}

3. Developer Experience
Frictionless provides libraries in multiple languages with consistent APIs:

# Python
from frictionless import validate, Package
package = Package('datapackage.json')
report = validate(package)

# Validation happens automatically
if report.valid:
    print("Data package is valid!")
else:
    for error in report.errors:
        print(f"Error: {error.message}")

// JavaScript
import { Package } from 'frictionless-js'

const package = await Package.load('datapackage.json')
const report = await package.validate()

if (report.valid) {
  console.log('Data package is valid!')
} else {
  report.errors.forEach(error => {
    console.log(`Error: ${error.message}`)
  })
}

Real-World Applications

Government Portal Enhancement:

{
  "name": "city-budget-2024",
  "title": "City Budget 2024",
  "description": "Complete municipal budget data with departmental breakdowns",

  // Frictionless simplicity
  "resources": [
    {
      "name": "budget-summary",
      "path": "budget-summary.csv",
      "schema": "budget-schema.json"
    }
  ],

  // Easy mapping to government standards
  "custom": {
    "dcat": {
      "theme": "http://publications.europa.eu/resource/authority/data-theme/GOVE",
      "accrualPeriodicity": "annual"
    },
    "project-open-data": {
      "bureauCode": ["019:20"],
      "programCode": ["019:006"]
    }
  }
}

Enterprise Data Catalog:

{
  "name": "customer-analytics-datasets",
  "title": "Customer Analytics Data Collection",
  "description": "Curated datasets for customer behavior analysis and segmentation",

  "resources": [
    {
      "name": "customer-profiles",
      "path": "s3://data-lake/customer-profiles/",
      "format": "parquet",
      "schema": "schemas/customer-profile.json"
    },
    {
      "name": "purchase-history",
      "path": "s3://data-lake/purchases/",
      "format": "parquet",
      "schema": "schemas/purchase-history.json"
    }
  ],

  // Enterprise-specific metadata alongside standard fields
  "custom": {
    "governance": {
      "dataOwner": "customer-analytics@company.com",
      "classification": "internal-confidential",
      "retentionPeriod": "7-years"
    },
    "technical": {
      "refreshSchedule": "daily-2am",
      "sourceSystem": "customer-db-prod",
      "qualityChecks": ["completeness", "accuracy", "timeliness"]
    }
  }
}

Choosing Your Standard: A Decision Framework

With multiple standards available, how do you choose the right approach for your organization? The answer often isn't picking one standard, but rather understanding how to layer them effectively.

The Progressive Standards Approach

Layer 1: Dublin Core Foundation
Start with Dublin Core's 15 elements for any metadata initiative:

# Minimum viable metadata - Dublin Core
title: "Quarterly Sales Report Q2 2023"
creator: "Sales Analytics Team"
subject: "sales, revenue, performance"
description: "Comprehensive sales performance data for Q2 2023 including regional breakdowns"
publisher: "Acme Corporation"
date: "2023-07-15"
type: "Dataset"
format: "CSV"
language: "en"
rights: "Internal use only"

Layer 2: Domain-Specific Standards
Add specialized vocabulary based on your sector:

Government/Public Data → DCAT:

# Dublin Core + DCAT for government
title: "Municipal Budget 2024"
description: "Complete city budget with departmental allocations"
publisher: "City Finance Department"

# DCAT additions
theme: "Government Finance"
accrualPeriodicity: "annual"
spatial: "Springfield, IL"
contactPoint:
  fn: "Budget Office"
  hasEmail: "budget@springfield.gov"
distribution:
  - downloadURL: "https://data.springfield.gov/budget-2024.csv"
    mediaType: "text/csv"

Research/Academic → Dublin Core + DDI:

# Dublin Core + research extensions
title: "Climate Change Perception Survey 2023"
creator: "Dr. Sarah Johnson, Climate Research Institute"
description: "National survey on public perception of climate change"

# Research-specific additions
methodology: "Random digit dialing telephone survey"
sampleSize: 2847
responseRate: 34.2
fundingSource: "National Science Foundation Grant #NSF-1234567"
ethicsApproval: "IRB-2023-045"

Layer 3: Technical Implementation
Choose implementation format based on your technical needs:

Web-based Discovery → Schema.org:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Dataset",
  "name": "Municipal Budget 2024",
  "description": "Complete city budget with departmental allocations",
  "url": "https://data.springfield.gov/budget-2024"
}
</script>

Developer-Friendly → Frictionless Data:

{
  "name": "municipal-budget-2024",
  "title": "Municipal Budget 2024",
  "description": "Complete city budget with departmental allocations",
  "resources": [
    {
      "name": "budget",
      "path": "budget-2024.csv",
      "schema": "budget-schema.json"
    }
  ]
}

Decision Factors

1. Organizational Context

Government/Public Sector:

Primary choice: DCAT (often with regional extensions)
Web presence: Add Schema.org for SEO
Implementation: Consider Frictionless for technical flexibility

Academic/Research:

Foundation: Dublin Core
Discipline-specific: DDI (social science), DataCite (research data), MODS (libraries)
Implementation: Frictionless for data packages, Schema.org for discoverability

Enterprise/Corporate:

Start with: Dublin Core basics
Business integration: Custom extensions for governance, lineage, quality
Implementation: Frictionless for flexibility, Schema.org if public-facing

2. Technical Capabilities

High Technical Resources:

Can implement multiple standards simultaneously
Custom mapping between standards
Advanced validation and quality checking

Limited Technical Resources:

Start with Frictionless Data for simplicity
Use hosted platforms that handle standards compliance
Focus on content quality over technical sophistication

3. Interoperability Requirements

High Interoperability Needs:

Strict DCAT compliance for government portals
Schema.org for web discoverability
Clear mapping to partner organization standards

Internal Focus:

Prioritize business-specific metadata
Use standards as a foundation but customize heavily
Focus on integration with existing tools and workflows

Implementation Roadmap

Month 1-2: Foundation

# Minimum viable metadata
title: "Required - Human readable name"
description: "Required - 2-3 sentences explaining the data"
creator: "Required - Who created this data"
date: "Required - When was this created/published"
format: "Required - File format (CSV, JSON, etc.)"

Month 3-6: Standardization

# Add standard vocabulary
title: "Quarterly Sales Performance Data"
description: "Revenue, units sold, and performance metrics by product category and region"
creator: "Sales Analytics Team, Acme Corp"
subject: ["sales", "revenue", "business intelligence"]  # Controlled vocabulary
type: "Dataset"  # Standard resource type
format: "CSV"
license: "Internal Use Only"  # Clear usage terms
identifier: "acme-sales-q2-2023"  # Unique ID

Month 6-12: Enhancement

# Full metadata with domain-specific extensions
title: "Quarterly Sales Performance Data Q2 2023"
description: "Revenue, units sold, and performance metrics by product category and region for Q2 2023 business analysis"

# Standard Dublin Core
creator: "Sales Analytics Team"
publisher: "Acme Corporation"
subject: ["sales", "revenue", "business intelligence", "performance metrics"]
type: "Dataset"
format: ["CSV", "JSON"]
date: "2023-07-15"
language: "en"
rights: "Internal Use - Business Confidential"

# Business-specific extensions
businessDomain: "Sales Analytics"
dataOwner: "VP Sales Operations"
refreshFrequency: "Monthly"
qualityScore: 94.7
approvalStatus: "Approved for Business Use"
retentionPeriod: "7 years"
sourceSystem: "Salesforce CRM"

The Future of Metadata Standards

The metadata standards landscape continues evolving, driven by new technologies, changing organizational needs, and lessons learned from large-scale implementations.

Emerging Trends

1. AI and Machine Learning Integration
Modern data catalogs increasingly use AI to enhance metadata:

# AI-enhanced metadata example
title: "Customer Transaction Data Q2 2023"
description: "Payment processing data for customer analytics"

# Traditional metadata
format: "CSV"
size: "2.3 GB"
records: 1847293

# AI-generated additions
aiGenerated:
  qualityScore: 94.2
  completenessProfile:
    customerID: 100%
    transactionAmount: 99.8%
    merchantCategory: 87.3%
  dataProfile:
    numericalColumns: 12
    categoricalColumns: 8
    dateColumns: 3
  suggestedTags: ["ecommerce", "payments", "customer-behavior"]
  similarDatasets: ["customer-profiles-2023", "payment-methods-analysis"]
  potentialIssues: ["Some merchant categories missing", "Date format inconsistency in 0.2% of records"]

2. Real-time and Dynamic Metadata
Static metadata gives way to dynamic, real-time information:

dataset:
  title: "Live Traffic Sensor Data"
  description: "Real-time traffic speed and volume from city sensors"

  # Static metadata
  publisher: "Department of Transportation"
  license: "CC-BY-4.0"

  # Dynamic metadata (updated automatically)
  lastUpdated: "2023-06-29T14:23:47Z"
  recordCount: 45729841  # Updates in real-time
  dataFreshness: "3 seconds"  # How recent is the latest data
  systemStatus: "operational"
  avgUpdateFrequency: "every 30 seconds"
  currentDataRange: "2023-06-29T14:20:00Z to 2023-06-29T14:23:00Z"

3. Privacy and Governance Integration
Metadata standards evolve to include privacy and compliance information:

dataset:
  title: "Customer Analytics Dataset"
  description: "Customer behavior and preference data for personalization"

  # Traditional metadata
  publisher: "Marketing Analytics Team"
  format: "Parquet"

  # Privacy-aware metadata
  privacyLevel: "PII-containing"
  gdprCompliant: true
  dataSubjects: "EU and US customers"
  legalBasis: "Legitimate interest for service improvement"
  retentionPeriod: "2 years"
  anonymizationMethod: "k-anonymity with k=5"
  sensitiveFields: ["customer_id", "email_hash", "location_zip"]
  accessControls:
    - role: "Marketing Analysts"
      permissions: ["read", "aggregate"]
    - role: "Data Scientists"
      permissions: ["read", "model"]
  auditLog: "All access logged for 7 years"

Convergence and Interoperability

The future points toward greater convergence between standards rather than fragmentation:

Cross-Standard Mapping
Tools and platforms increasingly support automatic translation between standards:

# Hypothetical future API
from metadata_converter import StandardsConverter

converter = StandardsConverter()

# Load data in any standard
metadata = converter.load('datapackage.json', format='frictionless')

# Convert to any other standard
dcat_output = converter.convert(metadata, target='dcat')
schema_org_output = converter.convert(metadata, target='schema.org')
dublin_core_output = converter.convert(metadata, target='dublin_core')

# Validate against multiple standards simultaneously
validation_report = converter.validate(metadata, standards=['dcat', 'schema.org'])

Universal Metadata APIs
Future data platforms may expose unified metadata APIs that work with any standard:

GET /api/metadata/dataset/12345?format=dcat
GET /api/metadata/dataset/12345?format=schema.org
GET /api/metadata/dataset/12345?format=frictionless

Recommendations for Future-Proofing

1. Build on Stable Foundations

Start with Dublin Core elements—they've remained stable for 25+ years
Use DCAT for data catalog applications—W3C backing provides long-term stability
Implement Schema.org for web presence—Google's support ensures longevity

2. Design for Flexibility

Use JSON-based formats for easier parsing and transformation
Implement clear separation between core metadata and extensions
Plan for automated migration between standards

3. Embrace Tooling

Invest in metadata management platforms that support multiple standards
Use validation tools to ensure quality and compliance
Implement automated metadata generation where possible

Conclusion

Navigating the metadata standards landscape doesn't require choosing a single "winner"—it requires understanding how different standards work together to solve different problems. Dublin Core provides the universal foundation, DCAT adds data catalog sophistication, Schema.org enables web discovery, and Frictionless Data offers practical implementation flexibility.

The key insight is that metadata standards should serve your organizational goals, not constrain them. Start with basic Dublin Core elements, add domain-specific standards as needed, and use tools like Frictionless Data to bridge the gap between standards and practical implementation.

Most importantly, remember that the best metadata standard is the one that gets used consistently. A simple, well-maintained Dublin Core implementation beats a complex DCAT setup that no one maintains. Focus on creating metadata that serves your users' discovery needs, and let standards provide the framework for consistency and interoperability.

As the data ecosystem continues to mature, organizations that invest in flexible, standards-based metadata approaches will find themselves better positioned to share data, collaborate across boundaries, and adapt to new technological developments. The metadata standards landscape may seem complex, but with the right approach, it becomes a powerful foundation for making data truly discoverable across organizations.

Whether you're building a government open data portal, an academic research repository, or an enterprise data catalog, the principles remain the same: start simple, build consistently, and let standards amplify the value of your data through improved discovery and interoperability.

How Rich Metadata Powers Data Discovery in Modern Data Catalogs

Anuar Ustayev — Wed, 25 Jun 2025 00:00:00 +0000

Introduction

In our previous post, we explored what metadata is and how it transforms raw data files into understandable resources. But metadata's true power emerges when you're managing not just one CSV file, but hundreds or thousands of datasets in a data catalog or portal.

Imagine you’re a researcher looking for climate data about New York City in a government data portal with 5,000 datasets. Without good metadata and search capabilities, you’d need to browse through endless pages of cryptically named files. With rich metadata powering a search engine, you can simply type “climate data New York” and instantly find relevant datasets ranked by relevance. This doesn’t yet account for using natural language queries to discover data, which we’ll explore in upcoming posts on modern data catalogs and portals.

The Scale Challenge: From Dozens to Thousands of Datasets

When Browsing Breaks Down

Small data collections work fine with simple browsing. A research team with 20 datasets can organize them in folders or basic categories. Users can scan through everything and find what they need.

But data catalogs grow quickly:

Municipal open data portals: Often contain 50-2,000 datasets covering everything from parking violations to budget data.
Enterprise data catalogs: Can house 500-50,000 datasets across departments and systems.
Research repositories: May contain hundreds of thousands of datasets from different studies and institutions.

At this scale, browsing becomes impossible. Users need search capabilities—and search engines need rich, consistent metadata to deliver relevant results.

[!info] A common question:
Can our search engine query the actual data rows within tables?
Answer:
Not directly. Indexing every data row—especially for tables with millions of time-series records—would be prohibitively large and memory-intensive for engines like Elasticsearch. Instead, we index structured metadata (e.g., table schemas, column descriptions, data dictionaries) and generate salient “key points” or summaries. This approach keeps the search index lean and performant while still guiding users to the most relevant tables.

The Standardization Imperative

When you have thousands of datasets, inconsistent or insufficient metadata creates chaos:

// Inconsistent or insufficient metadata across datasets:
Dataset 1: "nyc_311_jan2023.csv"
Dataset 2: "New York City 311 Service Requests - January 2023.xlsx"
Dataset 3: "January 2023 NYC Citizen Complaints and Service Calls.pdf"

Search engines can't effectively match these three related datasets to a user query like "NYC 311 data" because the metadata varies wildly.

The solution is a metadata schema—a standardized list of metadata fields that all datasets must follow, a simple example would be:

# Example metadata schema
required_fields:
  - title          # Human-readable dataset name
  - description    # 2-3 sentence explanation
  - owner          # Dataset maintainer/publisher
  - license        # Usage permissions
  - last_updated   # When data was last modified

recommended_fields:
  - tags           # Keywords for discovery
  - category       # Thematic classification
  - format         # File format (CSV, JSON, etc.)
  - temporal_coverage  # Date range covered

optional_fields:
  - geographic_coverage  # Spatial extent
  - update_frequency    # How often data changes
  - data_quality_score  # Completeness/accuracy rating

This schema enables three critical capabilities:

Systematic indexing: Search engines can reliably index the same fields across all datasets, ensuring consistent search behavior.
Enforced quality: Required fields ensure every dataset has minimum discoverable information before publication.
Guided creation: Data publishers know exactly what metadata to provide, with clear distinctions between required, recommended, and optional fields.

[!info] Note about our simple yet powerful metadata standard and tooling
Frictionless Data is a progressive open-source framework for building data infrastructure – data management, data integration, data flows, etc. It includes various data standards and provides software to work with data.

Search Engines: The Discovery Engine Behind Data Catalogs

Popular Platform Combinations

Modern data catalogs pair with powerful search engines to handle discovery at scale:

CKAN + Apache Solr: CKAN uses Solr's faceted search and text analysis capabilities to index dataset metadata, enabling complex filtering by organization, tags, formats, and date ranges.
OpenMetadata + Elasticsearch: OpenMetadata leverages Elasticsearch's real-time indexing to make data assets searchable immediately after ingestion, with support for complex queries across schemas, lineage, and usage patterns.
Custom portals + various engines: Many organizations build custom data portals using search engines like Elasticsearch, Solr, or even newer solutions like Meilisearch for specific performance requirements.

How Search Engines Index Metadata

Here's what happens when you add a new dataset to a catalog:

Ingestion: The catalog system reads both built-in metadata (filename, format, size) and external metadata files (title, description, tags)
Parsing & Validation: Metadata is parsed according to the catalog's schema and validated for required fields
Transformation: Data is normalized—dates converted to standard formats, tags split into arrays, categories mapped to controlled vocabularies
Indexing: The search engine creates searchable indexes from metadata fields, with different indexing strategies for text search vs. faceted filtering
Ranking: Search algorithms determine relevance based on text matching, metadata completeness, dataset popularity, and recency

Rich Metadata = Better Matches

Title & Description: The Foundation of Text Search

Compare these two approaches to naming the same dataset:

Minimal metadata approach:

filename: NYC-311-2023-Q1.csv
title: [same as filename]
description: [empty]

Rich metadata approach:

filename: NYC-311-2023-Q1.csv
title: New York City 311 Service Requests - Q1 2023
description: Comprehensive dataset of citizen complaints, service requests,
and municipal responses from NYC's 311 system during January-March 2023.
Includes request types, geographic distribution, response times, and
resolution status for over 500,000 service requests.

When someone searches for "NYC citizen complaints," the rich metadata provides multiple text matching opportunities:

"NYC" matches "New York City"
"citizen" matches "citizen complaints"
"complaints" matches the description text
"service requests" provides additional context

The search engine can confidently rank this dataset highly for the user's query.

Tags & Keywords: Enabling Faceted Discovery

Tags transform search from simple text matching to structured exploration:

# Example rich tagging for a transportation dataset
tags:
  - transportation
  - public-transit
  - bus-routes
  - geographic-data
  - real-time
  - gtfs-format

These tags enable users to:

Filter results: "Show me only transportation datasets"
Discover related data: "Other datasets tagged 'gtfs-format'"
Refine searches: "Real-time transportation data"

Categories & Themes: Structured Browsing Paths

While search handles specific queries, categories provide structure for exploration:

Government Data Portal Categories:
├── Transportation
│   ├── Public Transit
│   ├── Traffic & Parking
│   └── Infrastructure
├── Environment
│   ├── Air Quality
│   ├── Water Resources
│   └── Climate & Weather
└── Demographics
    ├── Census Data
    ├── Housing
    └── Economic Indicators

Good categorization helps users discover datasets they didn't know existed.

Temporal & Spatial Metadata: Context That Matters

Time and location metadata enable powerful filtering:

# Temporal metadata
date_created: "2023-01-15"
date_modified: "2023-03-20"
temporal_coverage_start: "2023-01-01"
temporal_coverage_end: "2023-03-31"
update_frequency: "daily"

# Spatial metadata
geographic_coverage: "New York City, NY, USA"
bounding_box:
  north: 40.9176
  south: 40.4774
  east: -73.7004
  west: -74.2591

This enables queries like:

"Show me datasets updated in the last week"
"Find data covering Manhattan from 2020-2023"
"Daily updated transportation data"

Real-World Example: Search in Action

Let's trace how rich metadata helps a user find relevant data:

User query: "climate data New York"

Dataset 1 (Poor metadata):

filename: weather_station_data_2023.csv
title: weather_station_data_2023
description: [empty]
tags: [empty]

Dataset 2 (Rich metadata):

filename: weather_station_data_2023.csv
title: New York State Climate Monitoring Network - 2023 Weather Data
description: Hourly temperature, precipitation, humidity, and wind measurements
from 45 weather stations across New York State. Data quality controlled and
validated by NYS Climate Office. Includes extreme weather events and monthly
climate summaries.
tags: climate, weather, temperature, precipitation, new-york, monitoring, hourly-data
geographic_coverage: New York State, USA
temporal_coverage: 2023-01-01 to 2023-12-31

Search engine matching process:

Text relevance: Dataset 2 matches "climate" (exact match in tags and description) and "New York" (in title and geographic coverage)
Metadata completeness bonus: Dataset 2 has rich descriptions and complete metadata fields, suggesting higher quality
Geographic precision: "New York State" closely matches user's "New York" query
Thematic relevance: Climate-related tags confirm this dataset is specifically about climate data

Result: Dataset 2 ranks much higher and provides the user with clear information about whether it meets their needs—all before they even download it.

Building for Discovery: Metadata Standards

Popular Metadata Schemas

Successful data catalogs adopt standardized metadata schemas:

DCAT (Data Catalog Vocabulary)

W3C standard for web-based data catalogs
Defines common properties: title, description, keywords, themes, publisher, license
Enables cross-catalog data discovery and harvesting

Schema.org Dataset

Structured data markup for web search engines
Helps Google and other search engines understand and index your datasets
Includes properties for data downloads, temporal coverage, and spatial coverage

Frictionless Data Package

Lightweight standard with focus on data usability
Emphasizes clear resource descriptions and table schemas
Popular in research and scientific data communities

Custom Organizational Schemas

Extended versions of standards with domain-specific fields
Example: Adding "data_sensitivity_level" for enterprise catalogs
Balance between standardization and specific organizational needs

The Consistency Advantage

The most important factor isn't which schema you choose—it's applying it consistently across all datasets. A catalog where every dataset has a clear title, description, tags, and category will outperform one with perfect but inconsistent metadata.

# Minimal but consistent schema across all datasets:
title: [required - human readable name]
description: [required - 2-3 sentences explaining the data]
tags: [required - 3-5 relevant keywords]
category: [required - from controlled vocabulary]
license: [required - usage terms]
last_updated: [required - ISO date format]

This consistency enables:

Reliable search results: Users can depend on finding complete information
Effective filtering: Faceted search works when fields are consistently populated
Automated processing: APIs and integrations can rely on standard fields being present

Conclusion

Rich metadata is the bridge between data creators and data users at scale. While a single CSV file might need only basic context, data catalogs with hundreds or thousands of datasets require standardized, searchable metadata to remain useful.

The combination of rich metadata and powerful search engines transforms data discovery from a frustrating browsing experience into targeted, relevant search results. Whether you're using CKAN with Solr, OpenMetadata with Elasticsearch, or building a custom solution, the principle remains the same: invest in consistent, descriptive metadata and your users will find exactly what they need.

In our next post, we'll dive into the practical aspects of implementing metadata standards in your data portal—covering schema design, validation workflows, and migration strategies for existing catalogs.

Basics of Metadata: How It Helps Understand Your Data

Anuar Ustayev — Mon, 02 Jun 2025 00:00:00 +0000

Introduction

In the age of big data, raw datasets alone are seldom enough. Without context, understanding a table’s fields, provenance, or usage constraints can be a guessing game. Metadata—often described as “data about data”—fills this gap by providing structured, machine-readable descriptions of your datasets. In this article, we’ll explore:

What metadata is, at a technical level
Core metadata concepts including built-in information and external metadata.
Basics of further ingestion of metadata in various systems.

Whether you’re building an internal data catalog or a public open-data portal, a robust metadata strategy is the key to making data discoverable, interoperable, and reusable.

1. Defining Metadata

Metadata is information that describes the characteristics and context of a data asset. It bridges the gap between raw data and meaningful insight by answering questions such as:

What the data represents (descriptive metadata)
How it is structured or formatted (structural metadata)
Who, when, and where the data was created or modified (administrative metadata)

At the simplest level, metadata can be built into your storage layer (e.g., file name, format, timestamps). For richer context, you can supply external metadata files (JSON, YAML, XML, or simple text) that complement and enhance built-in attributes.

2. From Built-In to External Metadata: A Step-by-Step Example

Let’s say you have a simple CSV file of temperature readings:

// File on disk:
temperature-readings-20250528.csv

// Content preview:
timestamp,value
2025-05-28T00:00:00Z,15.2
2025-05-28T01:00:00Z,14.9
...

2.1. Built-In Metadata

Every file system (and HTTP response) carries minimal metadata. For our CSV:

File name: temperature-readings-20250528.csv
Media type: text/csv

These two attributes already tell machines:

what the file is called, and
how to parse it (as comma-separated values).

But to your users—or to a search engine—you need richer context.

[!info] Note about other built-in metadata:
Beyond file name and media type, file systems and delivery protocols expose several other built-in metadata attributes you can leverage before you even add an external metadata file:

File System Attributes

Size (Content-Length in HTTP): the total byte length of the CSV.

Timestamps:

Created (birthtime on many Unix-style systems)

Last modified (mtime)

Last accessed (atime)

Permissions & Ownership: who owns the file and its read/write/execute bits.

HTTP/Transport Metadata (when serving over HTTP)

ETag or Last-Modified header: for cache validation.

Content-Encoding: if you gzip/brotli the CSV in transit.

Content-Disposition: suggests a download filename or whether to display inline.

CSV-Specific “Built-In” Info (derivable by quickly inspecting the file)

Header Row: column names—essential structural metadata.

Row Count: number of records (you can compute this cheaply on ingest).

Character Encoding: e.g. UTF-8 vs. ISO-8859-1 (often exposed as charset in the MIME type).

2.2. External Metadata

Often, file names and media types alone are insufficient to convey the context users need. To enrich your CSV with human-readable information, you can provide an external metadata file. For a single attribute—such as a title—you might start with a basic text file named metadata.txt:

Hourly temperature readings for May 28, 2025

While this simple file supplies a clear title, real-world use cases typically require multiple metadata fields (e.g., title, description, type). To represent these as structured data that programs can parse reliably, a key-value format is ideal. Consider the following YAML example:

title: Hourly temperature readings for May 28, 2025
description: |
  This CSV contains one-hour-interval temperature measurements
  recorded in Almaty, Kazakhstan on May 28, 2025. All timestamps
  are in UTC. Data sourced from XXX weather station.

[!info] What is YAML?
YAML ("YAML Ain't Markup Language") is a human-friendly serialization format. Its indentation-based syntax and key-value structure make it easy to author and parse, making YAML a popular choice for metadata, configuration, and data interchange.

The same information rendered in tabular form:

Metadata Field	Value	Purpose
`title`	Hourly temperature readings for May 28, 2025	Provides a concise, human-readable name—easier for users to scan than a raw filename
`description`	This CSV contains one-hour-interval temperature measurements recorded in Almaty, Kazakhstan on May 28, 2025. All timestamps are in UTC. Data sourced from XXX weather station.	Offers detailed context upfront, improving relevance judgment without inspecting data

[!info] Why add external metadata?
Providing even a simple companion metadata file gives users immediate context—transforming opaque filenames into understandable resource descriptions. It enhances discoverability, improves usability, and lays the groundwork for rich search and filtering features.

2.3. Taking It Further: Additional Attributes

Once you’ve mastered title and description, you can add:

keywords: e.g. temperature, Almaty, time series
license: e.g. CC-BY-4.0
provenance: who collected it and when (e.g. collectedBy: XXX weather station).

Each new attribute is just another line in your metadata file:

keywords: temperature,Almaty,time series
license: CC-BY-4.0
collectedBy: XXX weather station

Many systems would automatically pick these up—no custom code required.

2.4. How Ingestion Works

Discovery
- An ingestion pipeline identifies data files and their companion metadata files (e.g., .meta.txt, .json, or .yaml).
Parsing
- Read built-in metadata attributes (filename, media type, file size, timestamps) directly from the file system or HTTP headers.
- Parse the external metadata into structured fields (e.g., title, description, keywords).
Validation
- Validate parsed metadata against a predefined schema or set of rules to ensure completeness and correctness.
Transformation
- Normalize and transform metadata values (e.g., convert date formats, split comma lists into arrays).
Indexing
- Store both built-in and external metadata in a search or catalog index, mapping specific fields to facets, full-text search, and structured exports (JSON‑LD, DCAT).
- This index powers discovery features like faceted filters, semantic ranking, and metadata exports.

2.5. Why This Matters

Clarity for Users: A friendly title beats a cryptic filename.
Better Search: Rich text in description fuels full-text relevancy ranking.
Faceted Filters: Adding keywords and license turns those into clickable filters.
Automation: Keep your metadata alongside your data; pipelines stay simple.

By starting with two built-in attributes and a tiny external text file, you can already transform raw CSV dumps into a fully searchable, self-documenting dataset. From here, extending to full JSON-LD or DCAT becomes a natural next step.

Conclusion

Implementing metadata is not just a best practice—it's essential for unlocking the value of your data. By combining built-in attributes (file name, media type, timestamps) with external metadata files, you provide the context users need to discover, understand, and trust your datasets. A clear metadata strategy powers faceted search, semantic ranking, and linked-data features that scale from a simple CSV repository to enterprise-grade data catalogs. Start small with human-readable titles and descriptions, then grow your metadata alongside your data products to build truly data-driven experiences.

Why We Decoupled CKAN’s Frontend — And What We Gained with PortalJS

Anuar Ustayev — Wed, 28 May 2025 00:00:00 +0000

The short version?
CKAN is a rock-solid backend for open data portals.
But what happens when you want to modernize the user experience?
For us, the answer was simple: decouple the frontend.

Modern open data portals need more than just a reliable backend. They need speed, customization, and a frontend that’s easy to scale and maintain.

CKAN remains one of the most trusted open-source backends for managing datasets. But when it comes to the frontend experience, we chose to decouple — and here’s why.

In this article, we’ll share why we chose to separate the frontend from CKAN’s traditional stack, how we approached it with PortalJS, and what we learned about performance, scalability, and developer experience along the way.

CKAN: A Powerful Engine for Open Data

CKAN is widely used by governments, NGOs, and research institutions. Its backend is flexible, reliable, and API-driven — perfect for managing datasets at scale.

But CKAN ships with a built-in frontend based on Flask, a lightweight Python web framework. While this works well for many use cases, we found that as user expectations evolved, so did the need for a more dynamic, scalable, and customizable frontend experience.

That’s when we started asking:

What if we didn’t replace CKAN, but extended it — with a modern frontend layer built around today’s web standards?

What Is PortalJS and How Does It Work with CKAN?

PortalJS is a decoupled frontend framework built on Next.js. It connects to CKAN via API and replaces the traditional Flask frontend with a faster, more flexible layer.

Key Features

Static Site Generation (SSG): Pre-render content that doesn’t change often
Incremental Static Regeneration (ISR): Update static content in the background
Serverless functions: Handle dynamic parts on demand
React + TailwindCSS: Build beautiful, responsive interfaces

This means you can keep CKAN’s backend and get a frontend that:

Loads faster
Scales easier
Is more fun to build on

The Case for a Decoupled Frontend

Our decision wasn’t about what's wrong with CKAN — it was about what's possible with a decoupled architecture.

Monolithic Frontends: Solid, But Not Always Flexible

The traditional CKAN frontend is tightly coupled to the backend:

Each page is rendered dynamically by the server
Customizing the UI often means diving into Python templates
Developers need to manage full local stacks (Docker, Solr, PostgreSQL, Redis)

This setup is reliable but comes with friction — especially for frontend teams used to modern JavaScript tooling and performance patterns.

Why We Decoupled: The Benefits of Using PortalJS with CKAN

We built PortalJS to bring the best of both worlds together:

Keep CKAN as the backend
Replace the frontend with a modern, decoupled layer powered by Next.js

PortalJS communicates with CKAN over its existing API. You still use CKAN to manage datasets, users, organizations, and metadata — but everything the user sees is rendered through a faster, more flexible frontend.

What We Gained by Decoupling

Better Performance

PortalJS uses Next.js features like:

Static Site Generation (SSG) for pages that rarely change
Incremental Static Regeneration (ISR) for fresh content, without backend load
Serverless functions for dynamic rendering only when needed

This reduces API strain, speeds up page loads, and improves SEO dramatically.

Happier Developers

Setting up CKAN’s frontend locally means spinning up Docker containers, databases, and search services.

With PortalJS, the workflow looks like this:

npm install
npm start

That’s it. Developers can focus on building features, not configuring infrastructure.

More Customization

Because PortalJS is built with React and TailwindCSS, it’s easy to:

Redesign pages and navigation
Add filters and search interactions
Create responsive, mobile-friendly layouts

You're no longer bound to the default CKAN UI — you can make it your own.

Example: How We Handle Dataset Metadata Pages

In CKAN, every metadata page is rendered dynamically — even though most of that information rarely changes. And every visit hits the backend and database.

With PortalJS, we pre-generate those pages using SSG. That means:

Instant load times
No database hit
SEO-friendly static content

For parts that do change (like download counts or updated timestamps), we use client-side rendering or ISR.

It’s the best of both worlds.

CKAN + PortalJS: A Collaborative Architecture

PortalJS is a frontend layer that works with CKAN — not against it.

You keep CKAN’s backend, workflows, extensions, and APIs.

But you gain:

A smoother user experience
Easier customization
Better performance and scalability
Simpler developer onboarding

It’s the same engine — just a better dashboard.

Who This Is For

We’ve seen PortalJS help:

Governments modernize their public open data portals
Nonprofits simplify dataset publishing
Enterprises create internal data platforms with custom branding and UX

If you're already using CKAN and want a more modern, maintainable frontend — this is for you.

Final Thoughts: Let CKAN Do What It Does Best

CKAN remains one of the best backend systems for open data.

PortalJS doesn’t change that — it simply opens new possibilities.

By decoupling the frontend, we’ve been able to deliver faster, more beautiful, more usable portals — while still relying on the CKAN core that’s trusted worldwide.

🔗 Explore the full technical breakdown

FAQs

What’s the main reason to decouple the CKAN frontend?

To improve performance, scalability, and developer experience by using a modern, API-first frontend.

Does PortalJS replace CKAN?

No. PortalJS replaces only the frontend. CKAN still powers the backend, APIs, and admin functions.

Can I use existing CKAN extensions?

Yes. Since the backend remains CKAN, your extensions continue working as expected.

Where do I deploy PortalJS?

It runs on any frontend hosting provider. We recommend Vercel, Netlify, or Cloudflare Pages.

What’s the dev experience like?

Fast. No Docker required. Just clone, install dependencies, and start building with React.

Why NASA — and Anyone Using CKAN — Should Consider a Decoupled Front-End with PortalJS

Anuar Ustayev — Mon, 19 May 2025 00:00:00 +0000

Imagine a Faster, Smarter NASA Data Portal

What if a climate scientist visiting data.nasa.gov could explore global time-series datasets through an interface that loads instantly, adapts to their workflow, and feels intuitive—on any device?
Today, NASA’s portal runs on CKAN’s built-in frontend, which serves its purpose but limits performance and flexibility. By adopting a decoupled frontend using PortalJS, NASA could dramatically improve speed, user experience, and accessibility—without changing the backend.
This shift unlocks a more modern open data experience, tailored for researchers, analysts, and developers working with large, complex datasets.

The Current CKAN-Based Approach

NASA’s portal uses CKAN’s built-in frontend — a unified structure where the user interface and backend are tightly connected.

Monolithic UI & API

CKAN’s integrated UI and backend deliver robust search and data-management workflows out of the box. It serves well for dataset CRUD, geospatial previews, and basic theming.

What Works:

Out-of-the-box dataset management
Geospatial previews
Basic theming and search

Limitations

Release Cycle Coupling: UI updates often require CKAN core upgrades or plugin rebuilds.

[!info] In simple terms
If you want to tweak the interface, you have to dig into the backend code. That slows things down and makes small changes more complicated than they should be.

Branding Constraints: Deep theming adjustments can be labor-intensive and risk conflicts with upstream.

[!info] In simple terms
Making your portal look like “NASA” (or any other organization) takes a lot of effort, and those changes may not survive future updates.

Performance Overheads: Server-side rendering for every page load can introduce latency under heavy traffic.

[!info] In simple terms
Every time someone opens a page, the server has to rebuild it. That slows things down when lots of people visit at once.

Why a Decoupled PortalJS Front-End?

PortalJS is a frontend framework purpose-built for CKAN. PortalJS consumes CKAN’s REST API and renders the frontend independently using Next.js. This architecture gives teams more control over UX, while keeping the backend untouched.

Key advantages:

1. Rapid Iteration

PortalJS applications consume CKAN’s REST APIs, enabling independent deployment of UI components. New features — like custom data-visualization panels — can ship without touching the CKAN codebase.

2. Enhanced UX & Branding

Component Library: Reusable “DatasetCard,” “FilterPanel,” and “MapExplorer” snippets conform to NASA’s design system.
Responsive & Accessible: Built-in patterns for ARIA compliance and mobile-first layouts ensure broad reach.

[!info] In simple terms
You can make the portal look and feel exactly how your users expect, with reusable building blocks. Works well on phones and meets accessibility standards by default.

3. Scalability & Resilience Scalability & Resilience By hosting the PortalJS front end on a CDN (e.g., Cloudflare) and separating it from CKAN servers, static assets load faster and UI downtime can be decoupled from backend maintenance windows.

[!info] In simple terms
Pages load almost instantly because they’re already generated and stored near your users — no waiting for the server to respond.

A Blueprint for Decoupled Architecture

A decoupled setup keeps CKAN as the backend and places PortalJS on top as the user-facing layer. The backend stays stable, while the frontend can evolve quickly and scale independently. You serve pages fast, keep your infrastructure lightweight, and give teams more control.

1. API Gateway:

AWS API Gateway or CloudFront Lambda@Edge enforcing API keys, rate-limits, and caching for CKAN REST endpoints.

2. PortalJS Application:

Deployed to an S3 bucket + CloudFront distribution.
Integrations: CKAN’s /api/3/action routes for dataset search, resource download, and user authentication.

3. Extensibility Hooks:

Widget Registry: Dynamically load visualization widgets (e.g., D3 timelines, Cesium 3D maps) based on dataset metadata.
Theme Manager: Allow NASA web teams to tweak color tokens and font scales via a JSON config file.

Why This Matters

CKAN’s backend is solid. But a modern frontend makes ALL the difference.

This isn’t just about NASA. The same constraints exist for many governments, research labs, and institutions using CKAN.

Decoupling with PortalJS unlocks:

Speed
Flexibility
Custom and modern UX
Resilient architecture

[!info] Bottom line
Your portal can grow, adapt, and look modern — without expensive rebuilds or risky changes to your data backend.

See It in Action

Curious what this would look like for your team? Start a pilot in minutes — no backend changes required.

If you’d like to explore a PortalJS pilot on your CKAN instance — government, research institution, or enterprise — visit our site and get started in minutes!

📩 Talk to us to discuss your use case.

The New Look of PortalJS Cloud

Anuar Ustayev — Tue, 01 Apr 2025 00:00:00 +0000

At PortalJS Cloud, we believe that managing and sharing open data should be effortless, beautiful, and accessible. That’s why we’ve launched a complete redesign of the PortalJS Cloud frontend—combining performance upgrades with a sleek, modern interface designed for everyone, from governments and non-profits to academic institutions and private organizations.

The new version offers a cleaner interface, enhanced accessibility, and lightning-fast search—making it the smartest choice for modern open data initiatives.

Let’s explore what’s new and how these improvements help you deliver a better data experience to your users.

What’s New?

A Revamped User Interface

We’ve introduced a fresh, professional design across the entire portal. Pages like the homepage, dataset view, and search experience now feel smoother and more intuitive for all users.

Modern, minimalist layout for a polished first impression:

Home page

Search page

Dataset page

Resource Preview Page

Optimized for performance
Fully responsive across desktop, tablet, and mobile devices
Designed with accessibility in mind, meeting WCAG 2.1 / 2.2 standards
- All interactive elements (filters, tabs, buttons) are keyboard-friendly
- “Skip to content” navigation for screen reader users

Smarter Search & Filtering 🔍

We’ve made the search experience much easier and more powerful—especially when it comes to filtering and sharing results.

Shareable search results

We’ve rebuilt the search page so that every filter you apply—such as keywords, tags, or formats—is automatically reflected in the page URL. This is known as URL-synced search state.

In other words, the portal keeps track of your search settings directly in the web address.

What this means for the user:

Let’s say a user filters for “climate data from 2024”. The page URL will automatically update to reflect those filters. They can now copy that link and send it to a colleague, who will land on the exact same filtered view. No need to repeat the search.

Plus, filters are saved in the browser history. So if someone clicks on a dataset, they can use the Back button to return to their filtered results—just like browsing a regular website.

You can copy and share the page link, and others will see the exact same filtered view
Saved URLs can act like bookmarks, helping users return to specific filtered searches anytime
The portal behaves more like modern apps, where your current view is always reflected in the URL

Minimal state usage

In web development, "state" is the information a page keeps track of while you’re using it—like which filters you’ve applied or what search terms you’ve entered.

We’ve made sure that the search page in PortalJS Cloud only stores the essential state. This means the system isn’t overloaded with unnecessary data, making everything simpler and faster behind the scenes.

What this means for the user:

Pages load faster
Filters respond instantly
The portal is more stable—even with large datasets
Less chance of bugs or crashes

Faster and more reliable behind-the-scenes logic with React Context (no more prop-drilling)

Previously, when different parts of a web page needed to "talk" to each other (for example, when a filter affected the search results), the data had to be passed through multiple layers—like a game of telephone. This process is called prop-drilling, and it can get messy and hard to manage.

Now, thanks to improved architecture (using something called React Context), all search filters and components stay in sync automatically—without needing to repeat the same logic in multiple places. Think of it like everyone checking the same live dashboard instead of passing sticky notes around.

What this means for the user:

The search page is easier to maintain, faster to load, and more robust
All filters, search bars, and results stay perfectly in sync
No lag, no glitches—just smooth, instant updates without reloading the page. The technology behind the scenes is optimized to deliver results quickly, even on slower internet connections.
New features and filters can be added quickly without breaking anything

These upgrades make it easier for users to find the data they need—and to share it with others.

Better Data Visualizations 📊

We’ve significantly improved the CSV preview component to make exploring data easier, more accessible, and more interactive.

Accessibility-First Design

The preview is now fully compliant with WCAG 2.1 / 2.2 standards, meaning it works smoothly with keyboard navigation and screen readers. Users can tab through rows and columns or use assistive technologies to understand and interact with the data.

New Interactive Features

We’ve added powerful new tools to help users make sense of large tables:

Search within the table – Instantly find specific columns or values. Example: type “GDP” and jump straight to that column or row.
Column selector – Choose which columns to show or hide. Great for reducing clutter when working with wide datasets.
Pagination control – Browse large datasets more easily, page by page.
Export to JSON – Download the data in developer-friendly format for further analysis or integration.

Form & Filter Accessibility

We’ve upgraded all form and filter components to be fully accessible (see Accessibility improvements screenshot above).

Accessible labels – Every input field, dropdown, and button now has a proper label that screen readers can detect and announce.
Live filter updates – When a user applies a filter, screen readers now receive real-time feedback about the change. For example, “3 datasets found for your selection” is announced automatically.

What this means for the user:

People using screen readers or keyboard navigation can interact with forms just as smoothly as others
Everyone can confidently use filters and forms, knowing exactly what’s happening on the page
Your data portal is more inclusive, user-friendly, and aligned with accessibility best practices

Visual & Semantic Structure

Beyond visual design, we’ve made deeper improvements to the way the site is structured under the hood (see Accessibility improvements screenshot above).

Verified color contrast – All UI elements now meet recommended contrast ratios, ensuring readability for users with low vision or color blindness.
Semantic HTML and ARIA roles – We’ve reorganized the site structure using proper tags (like headers, sections, and landmarks), and added ARIA roles to give screen readers clear context and navigation cues.

What this means for the user:

Text and buttons are easier to read in all lighting conditions
Assistive technologies can “understand” the layout and flow of each page
The portal feels more predictable and intuitive for everyone, including users with disabilities

Easy Customization, Delivered Fast

Want to match your data portal to your brand’s color and logo? No problem. We’ve made it incredibly simple for our developers to update key visual elements like:

Main template color – Choose a color that fits your brand
Portal logo – Upload your organization’s logo in seconds

These changes are configured via .env variable:

NEXT_PUBLIC_THEME_COLOR and NEXT_PUBLIC_PORTAL_LOGO

For example:

NEXT_PUBLIC_THEME_COLOR=#8847cd
NEXT_PUBLIC_PORTAL_LOGO=/images/logos/purple.jpg

What this means for you:

If you need a color or logo update, we can make the change and deploy it almost instantly—no long wait times, no messy custom coding. Because we’ve built it smart from the start.

Why Did We Change It?

The old template had some outdated design elements and usability gaps.
We wanted to ensure first-time users have a smooth experience setting up their portal.
Aligning with modern web standards makes the portal more maintainable and professional.
Compliance with accessibility standards was a priority (more on that in our accessibility article).

Why We Redesigned PortalJS Cloud

We didn’t just give PortalJS Cloud a facelift—we rebuilt it with purpose.

The previous design, while functional, had limitations. Some visual elements felt outdated, and usability gaps made it harder for new users to get started quickly.

So we took a step back and asked: What does a modern data portal need in 2025?

The answer was clear:

A clean, intuitive interface that welcomes first-time users
Streamlined performance across all devices
Built-in accessibility, ensuring everyone can navigate and explore data
A future-proof foundation that aligns with today’s web standards—and tomorrow’s

You can read a bit more about why compliance with accessibility standards were a priority in our accessibility article.

This redesign is about helping governments, non-profits, researchers, and companies launch professional, user-friendly data portals—faster and more efficiently than ever before.

👉 Try the new PortalJS Cloud today with a free 14-day trial—no credit card required. Experience the new look, lightning-fast performance, and modern accessibility, all in minutes.

Before & After Comparison

Homepage before

Homepage after

Search page before

Search page after

How to Reduce Data Portal Costs by 90% or More

Anuar Ustayev — Fri, 14 Mar 2025 00:00:00 +0000

What if the key to smarter, more transparent governance wasn’t spending more—but spending less? Too often, data portals are seen as unavoidable financial burdens, rather than an opportunity. But they don’t have to be. Your data portal can be a cornerstone of transparency, innovation, and better decision-making without draining your budget.

Proprietary solutions, while promising the world, often deliver a painful bill. Hidden fees, rigid systems, and vendor lock-in quietly inflate costs while limiting your ability to adapt. And for what? A tool that’s supposed to serve your organization ends up as a line item that’s tough to justify.

But what if it didn’t have to be this way? What if you could cut your data portal expenses by 90%—or more—every year without compromising on performance, compliance, or scalability? It’s not just a possibility; it’s a practical reality. Let’s break down where your money is going, and how a smarter approach can turn your data portal from a financial burden into a strategic asset.

Where Do the Costs Come From?

If you’ve ever stared at a bloated invoice from a data portal vendor, you’ve probably wondered: Where is all this money going? Here’s what you’re often paying for:

Licensing Fees: Proprietary solutions can charge six or even seven figures annually. And guess what? Those fees only grow as your data needs increase.
Customization Costs: Need your portal to match your organization’s branding or workflows? Many platforms charge extra for every tweak.
Maintenance & Hosting: Whether it’s managing servers, ensuring uptime, or resolving bugs, these costs add up fast.
Training & Support: Onboarding your team often comes at an additional cost, and premium support packages can be staggeringly expensive.

When you add it all up, it’s no wonder small towns, mid-sized cities, and even large governments struggle to justify the expense. But it doesn’t have to be this way.

The Open Source Alternative: Why CKAN Is the Gold Standard

CKAN, the open-source data portal solution has become the gold standard for governments, NGOs, and research institutions. Unlike proprietary systems, CKAN is free to use, infinitely customizable, and backed by a global community of developers. But—and this is a big one—while CKAN saves you licensing fees, it doesn’t eliminate all costs. You still need hosting, maintenance, and a user-friendly frontend to make it work for your audience.

That’s where solutions like PortalJS Cloud come into play. By combining CKAN’s power with modern, fully managed hosting and customizable frontends, you get the best of both worlds: open-source affordability and enterprise-grade performance.

How PortalJS Cloud Helps You Save

Let’s break down exactly how PortalJS Cloud can help you trim 90% (or more) from your annual data portal expenses.

No Licensing Fees
Most proprietary platforms lock you into expensive annual contracts. PortalJS Cloud? None of that. Since it’s built on CKAN, you’re free from those hefty licensing costs. You’re essentially only paying for the hosting and support you need—not for the privilege of using the platform.
Streamlined Maintenance & Hosting
Maintaining your own CKAN instance can be time-consuming and expensive, especially if you’re not a tech-heavy organization. PortalJS Cloud offers fully managed hosting, which means we handle the backend, updates, and troubleshooting for you. That’s one less thing for your IT team to worry about (or for you to outsource).
Scalable Pricing
Instead of a one-size-fits-all model, PortalJS Cloud scales to meet your needs. For small towns with limited budgets, plans start at just $99/month. Larger governments can choose plans with advanced features while still saving massively compared to proprietary options.
Customizable Frontend Without Extra Fees
Proprietary platforms often charge an arm and a leg for customization. PortalJS Cloud lets you tailor your data portal’s design and functionality without breaking the bank. Want your portal to reflect your city’s branding? Done. Need specific features for public-facing users? No problem.
Faster Deployment, Lower Costs
Some systems take months to set up, driving up costs with each passing day. PortalJS Cloud offers 5-minute deployment for basic configurations, so you can start sharing data (and seeing value) almost immediately.

A Quick Comparison: PortalJS Cloud vs. Proprietary Solutions

Feature	Proprietary Solutions	PortalJS Cloud
Annual Cost	$100,000+	Starting at $1,188
Customization Fees	High	Included
Vendor Lock-In	Yes	No
Compliance Standards	Yes	Yes
Deployment Time	Months	Minutes

A Closer Look at Value

When you’re not tied to a vendor’s roadmap or locked into their support contracts, you’re free to build the portal your community actually needs. Whether it’s improving data accessibility, expanding transparency efforts, or aligning your data strategy with your long-term goals, the shift to a smarter, more flexible solution creates space for meaningful progress.

More Than Savings: Future-Proofing Your Data Portal

It’s not just about saving money. It’s about investing in a system that grows with you. PortalJS Cloud combines CKAN’s open-source foundation with modern architecture, ensuring your portal is scalable, adaptable, and built to last. Compliance with standards like WCAG 2.1 AA and DCAT ensures accessibility and interoperability, while features like automated updates and robust APIs simplify your workflow.

Whether you’re managing a small-town transparency initiative or a large-scale federal data program, the combination of CKAN’s open-source foundation and PortalJS’s modern capabilities ensures your portal stays fast, flexible, and future-ready.

By stepping away from rigid, high-cost systems, you regain control over your portal’s direction. This isn’t just a cost decision; it’s a strategic one.

Get Started Today

Your data portal doesn’t need to be a burden. With PortalJS Cloud, you can achieve a balance between affordability, functionality, and long-term viability.

Ready to cut your data portal costs by 90% (or more)? With PortalJS Cloud, you get the affordability of open source with the reliability of a fully managed platform.

💡 Start Your Free Trial today and see how easy and cost-effective your data portal can be. Or, if you’d prefer a closer look, schedule a demo with our team.

Enhancing CKAN Resource Uploads via API with Cloudflare R2, Workers, and Queues.

Anuar Ustayev — Wed, 26 Feb 2025 00:00:00 +0000

Introduction

CKAN, a powerful open-source data management system, traditionally stores only metadata about uploaded resources, leaving file storage to external solutions. In our implementation, we used Cloudflare R2 for resource storage, enabling users to upload assets either through a frontend application.

However, we lacked an API-based approach for users to upload files directly. This blog details how we solved this limitation by introducing a new API endpoint for resource uploads and integrating Cloudflare R2 with CKAN using Cloudflare Workers, Queues, and Notifications to automate metadata updates.

Problem Statement

By default, CKAN only stores metadata for resources, not the actual files. Users could:

Upload files through the frontend, which would generate and store the file URL in CKAN.
Provide an external URL directly via the UI or API.

However, there was no provision for users to upload images through the API while ensuring seamless metadata association.

Solution Approach

We tackled the issue in two steps:

Enhancing CKAN API to support pre-signed URLs for uploads.
Automating metadata linking after the upload using Cloudflare services.

Step 1: Modifying and Adding API Endpoints

We introduced two key API improvements:

resource_upload Endpoint: Returns a pre-signed URL for an existing resource, allowing users to upload assets directly.
Enhanced resource_create Endpoint: Generates a pre-signed URL whenever a new resource is created, enabling direct uploads.

Implementation

from ckanext.s3filestore.uploader import BaseS3Uploader
from ckan.plugins.toolkit import get_action
import ckan.plugins.toolkit as tk
import ckan.logic as logic
import logging

ValidationError = logic.ValidationError
log = logging.getLogger(__name__)

def _generate_presigned_url(resource_id):
    try:
        uploader = BaseS3Uploader()
        s3_client = uploader.get_s3_client()
        bucket_name = uploader.bucket_name
        object_key = f"resources/api_uploads/{resource_id}"

        params = {'Bucket': bucket_name, 'Key': object_key}
        presigned_url = s3_client.generate_presigned_url(
            ClientMethod='put_object', Params=params, ExpiresIn=3600)

        return presigned_url
    except Exception as e:
        log.error(f"Failed to generate presigned URL: {e}")
        raise ValidationError({"error": f"Failed to generate presigned URL: {e}"})

def resource_upload(context, data_dict):
    resource_id = data_dict.get('id')
    if not resource_id:
        raise ValidationError({"id": "Resource ID is required to generate upload URL."})

    get_action('resource_show')(context, {'id': resource_id})
    presigned_url = _generate_presigned_url(resource_id)
    return {"presigned_url": presigned_url}

@tk.chained_action
def resource_create(up_func, context, data_dict):
    result = up_func(context, data_dict)
    resource_id = result.get("id")
    presigned_url = _generate_presigned_url(resource_id)
    result["presigned_url"] = presigned_url
    return result

This enhancement enables users to securely upload files via the API without needing to interact with the frontend.

Step 2: Automating Metadata Linking Using Cloudflare

Once the file is uploaded, we need to link the uploaded file’s metadata with its CKAN resource entry. While the frontend seamlessly handles this by storing the file URL during resource creation, our API-based approach required an additional step to update metadata post-upload.

To automate this process, we leveraged:

Cloudflare R2 Event Notifications
Cloudflare Queues
Cloudflare Workers

Why Use These Cloudflare Features?

Notifications: Detect file uploads in real-time.
Queues: Buffer file upload events and handle them asynchronously.
Workers: Process queued events and update CKAN accordingly.

Implementation

Configuring R2 Event Notifications

Events triggered for objects in resources/api_uploads/ on PutObject, CopyObject, and CompleteMultipartUpload.
Sends an event to the r2-file-uploads queue.

Processing Events with Cloudflare Worker

export default {
  async queue(batch, env, ctx) {
    for (const message of batch.messages) {
      try {
        let data = typeof message.body === "string" ? JSON.parse(message.body) : message.body;
        console.log("Received R2 Event:", data);

        const objectKey = data?.object?.key;
        const resourceId = objectKey.replace("resources/api_uploads/", "");
        const fileUrl = `https://blob.datopian.com/${objectKey}`;

        console.log("Updating CKAN with URL:", fileUrl);

        // Fetch metadata (Content-Type)
        const metadataResponse = await fetch(fileUrl, { method: "HEAD" });
        if (!metadataResponse.ok) continue;

        const fileType = metadataResponse.headers.get("Content-Type") || "application/octet-stream";
        console.log("Detected MIME Type:", fileType);

        // Update CKAN resource metadata
        const ckanApiUrl = "https://demo.dev.datopian.com/api/3/action/resource_update";
        const ckanApiKey = env.CKAN_API_KEY;

        const updatePayload = { id: resourceId, url: fileUrl, format: fileType };
        const response = await fetch(ckanApiUrl, {
          method: "POST",
          headers: { "Content-Type": "application/json", "Authorization": ckanApiKey },
          body: JSON.stringify(updatePayload)
        });

        if (!response.ok) console.error("CKAN API Error:", await response.text());
        else console.log("CKAN update successful!");
      } catch (error) {
        console.error("Error processing message:", error);
      }
    }
  }
};

The provided JavaScript code is a Cloudflare Worker that processes events from a Cloudflare Queue triggered by file uploads to Cloudflare R2 storage. Here's a brief explanation:

Event Processing: The Worker listens for batch messages from the r2-file-uploads queue, which contains details about uploaded files.
Extracting Metadata: It extracts the uploaded file's key (path) and derives the resourceId from it.
Fetching File Type: It makes a HEAD request to the file URL to determine the MIME type.
Updating CKAN: It sends an API request to CKAN to update the resource metadata (file URL and format).
Error Handling: Logs errors if any step fails.

This automates linking uploaded files to CKAN resources, ensuring seamless metadata updates.

Future Possibilities

This automation can be extended to other CKAN functionalities, such as:

Uploading organization or group images via API.
Supporting other cloud providers (AWS S3, Azure Blob Storage) using their event-driven tools like AWS Lambda and Azure Functions.

Example Integrations

AWS S3 → SNS + SQS + Lambda → CKAN API

A file is uploaded to an S3 bucket.
An event notification triggers an SNS topic.
The SNS topic sends a message to an SQS queue.
A Lambda function reads the SQS message and updates the CKAN API with the file metadata.

Azure Blob → Event Grid + Functions → CKAN API

A file is uploaded to an Azure Blob Storage container.
Event Grid detects the upload and triggers an Azure Function.
The Azure Function extracts metadata and updates the CKAN API accordingly.

These platforms offer similar event-driven mechanisms, making it easy to integrate CKAN with different cloud providers.

Conclusion

By integrating CKAN APIs with Cloudflare R2, Workers, and Queues, we successfully:

Enabled direct file uploads via API.
Automated the linking of uploaded files to CKAN resources.
Leveraged event-driven processing to ensure seamless user experience.

This approach reduces API complexity for users while maintaining data consistency across systems. With similar integrations, CKAN can be extended to other cloud storage providers, enabling flexible and scalable data management.

Effortless User Management in PortalJS

Anuar Ustayev — Tue, 28 Jan 2025 00:00:00 +0000

Key Takeaways
PortalJS has made user management much smoother. Admins can now add existing users to organizations instantly by entering their email—no redundant invites, no errors. If the user isn’t in the system, the invite process works as usual. This update streamlines team management, reduces admin workload, and ensures a better onboarding experience.

PortalJS Now Supports Smoother User Management

Managing users across multiple organizations should be simple, not a tedious task. Yet, until now, adding existing users to a new team in PortalJS required unnecessary steps, redundant invitations, and, at times, frustrating errors.

The latest PortalJS user management update makes this process more intuitive. Now, admins can add existing users to an organization by simply entering their registered email in the Members tab. If the user already exists, the system assigns them instantly. No extra steps. No unnecessary complications. If not, the usual invite process applies. This ensures a smooth and efficient team management for data platforms.

What’s Improved?

Previously, the system would always attempt to create a new user when sending an invite, causing errors if the user was already registered. The update refines the user_invite function by:

Checking for existing users before sending an invite.
Assigning users instantly if they are already in the system.
Falling back to the traditional invite process only if necessary.
Improving error handling so issues with email invites don’t block the process.

These refinements remove friction, ensuring admins spend less time on user management and more time on their actual work.

A More Streamlined Onboarding Experience

For businesses and organizations scaling their data portals with PortalJS, efficient user administration is critical. This update reduces administrative workload for PortalJS admins, allowing them to focus on managing data, not troubleshooting user invites.

With these improved user management features in PortalJS, teams can collaborate more effectively, ensuring smooth workflows for data-driven organizations.

Get Started with PortalJS Cloud Today

Launch your Data portal in under 5 minutes with PortalJS Cloud—the fastest and simplest way to share, manage, and collaborate on data. No infrastructure setup, no complex configurations—just a streamlined solution for governments, non-profits, academics, and companies of all sizes.

Get started now and bring your data to life effortlessly.

Making PortalJS Cloud Admin Panel Accessible: The Digital Ramp Everyone Deserves

Anuar Ustayev — Sun, 26 Jan 2025 00:00:00 +0000

The Need for Accessibility: Building a Digital City for All

Imagine a city where half the streets have potholes so deep that some people can’t pass through. Where doors swing open only for those who fit a certain mold. Where street signs disappear when you look at them from a different angle. Sounds frustrating, right? That’s exactly how the web feels for millions of users when accessibility isn’t a priority.

Accessibility isn’t a nice-to-have—it’s the foundation of a web that welcomes everyone. At PortalJS Cloud, we didn’t just tick compliance checkboxes; we built a digital ramp that ensures everyone gets through the front door.

And we didn’t stop at the admin panel. Every data portal built with PortalJS Cloud—in just five minutes—is fully WCAG-compliant. Because accessibility isn’t a checkbox—it’s a mindset woven into everything we build.

WCAG 2.1 vs. WCAG 2.2: The Blueprint for Digital Inclusion

Think of WCAG (Web Content Accessibility Guidelines) as an evolving map for a fairer internet.

WCAG 2.1 (2018) was like upgrading a city’s infrastructure to accommodate more people—introducing smoother sidewalks, better-lit streets, and accessible public transport for those with diverse needs.
WCAG 2.2 (2023) took it further by fine-tuning details that improve usability for people with cognitive disabilities and motor impairments. Think of it as adjusting traffic lights, widening sidewalks, and ensuring door handles work for everyone.

The Journey to an Accessible Admin Panel

1. Auditing and Identifying Issues: Finding the Potholes

Before fixing anything, we needed to see where the cracks were. Using industry-standard tools like WAVE, axe DevTools, and Lighthouse, we ran a full audit. Every missing label, poor contrast ratio, and keyboard trap was a pothole we needed to fill.

2. Keyboard Navigation: Every Click Should Have a Clear Path

For some users, a mouse is as useful as a submarine in a desert. They navigate using keyboards, screen readers, or voice commands. So we made sure that:

Every button, link, and form could be reached using only a keyboard.
A clear visual focus was added to show users where they were on the page.
We built skip navigation links, the express lanes of web browsing, allowing users to jump straight to important sections.
All interactive elements were designed with a minimum target size of 24x24 pixels, ensuring ease of use for users with motor impairments.

3. Fixing Forms: Because Guesswork Shouldn’t Be a Requirement

Forms are the bureaucratic paperwork of the web. Done wrong, they become a maze. Done right, they feel like a conversation. We ensured that:

Every form field had clear labels and instructions.
Error messages were descriptive and polite, guiding users instead of scolding them.
Focus didn’t disappear like a magician’s trick—it stayed predictable and logical.
Error prevention mechanisms were implemented for sensitive data entries, giving users confirmation prompts or undo options.

4. Color Contrast: Making Text Stand Out Like a Lighthouse

Imagine reading a book where the ink is only a shade darker than the page. That’s what poor color contrast feels like. We ensured that:

Text and interactive elements had a minimum contrast ratio of 4.5:1, making them readable in all lighting conditions.
Alternative visual indicators (icons, underlines, patterns) helped those with color blindness navigate effortlessly.

5. Using Semantic HTML and ARIA Roles: Speaking a Universal Language

Screen readers rely on structure. If a webpage isn’t built with semantic HTML, it’s like trying to read a scrambled recipe. We:

Used semantic HTML to ensure that screen readers understood the page structure.
Added ARIA roles and attributes so users with assistive tech could grasp the purpose of each section.
Ensured consistent navigation elements throughout the admin panel to maintain familiarity for users.

6. Testing with Real Users: Because Theory is Nothing Without Practice

Automated audits are useful, but they don’t catch everything. So we tested the admin panel with:

Screen readers (NVDA, JAWS, VoiceOver) to ensure content made sense.
Keyboard-only navigation to confirm smooth interactions.
Real users with disabilities, because lived experience trumps theory every time.

Why Accessibility is a Competitive Advantage

This isn’t just about being compliant—it’s about creating an exceptional product. Here’s why accessibility matters:

Inclusivity is Innovation — The best designs work for everyone. Curb cuts, originally made for wheelchair users, are now a blessing for cyclists and parents with strollers. Digital accessibility follows the same rule.
SEO Rewards Accessibility — Search engines love structured, accessible content. Fixing accessibility often improves rankings.
Legal Risks are Real — Many companies have faced lawsuits for failing to meet accessibility standards. Avoiding them is smart business.
Better UX for Everyone — Accessibility improvements make life easier for all users—whether they’re on mobile, in a rush, or in a low-bandwidth area.

Accessibility Beyond the Admin Panel: A Fully Compliant Data Portal

Accessibility isn’t a feature—it’s the foundation. That’s why every data portal built with PortalJS Cloud is fully WCAG-compliant right out of the box. In just five minutes, users can deploy a portal that:

Provides seamless screen reader support, ensuring datasets are easily navigable.
Offers fully accessible data preview tables, making data exploration effortless for everyone.
Maintains optimized color contrast and keyboard accessibility, enhancing usability across all devices.
Eliminates mandatory dragging movements, allowing users to interact without precision-based gestures.

In an upcoming article, we’ll dive deeper into how we designed our out-of-the-box data portal templates to be fully WCAG-compliant, ensuring that accessibility is baked into every dataset preview, every UI element, and every interaction.

Accessibility is a Journey, Not a Destination

Making the PortalJS Cloud Admin Panel accessible wasn’t a one-and-done project. It’s an ongoing commitment—like maintaining roads, updating maps, and making sure everyone can move freely in our digital city.

The internet is a vast landscape, but we get to decide what kind of world we build. At PortalJS, we’re choosing one where everyone has a way in.

Want to make your product accessible? Start with the official WCAG guidelines and build a web that works for all.

Why PortalJS is the Future of Decoupled Frontend for Data Portals

Anuar Ustayev — Fri, 06 Dec 2024 00:00:00 +0000

Introduction

In the world of (open) data portals, a seamless, fast, and scalable frontend is crucial for delivering a great user experience. As the amount of data grows and user interactions become more complex, traditional monolithic frontends can become bottlenecks. This is especially true for platforms like CKAN, where the default frontend, built with Flask, can struggle to handle the demands of modern data-rich applications.

This is where PortalJS comes in—a modern, decoupled frontend framework designed specifically for data portals. Built on top of NextJS, it brings the latest in web performance optimization to the world of open data. Unlike CKAN’s built-in frontend, PortalJS enables faster load times, more efficient use of APIs, and a far more flexible development environment.

One of the key features that sets PortalJS apart is its ability to leverage NextJS’s rendering strategies, such as Static Site Generation (SSG), Server-Side Rendering (SSR), and Incremental Static Regeneration (ISR). These techniques ensure that your data portal isn’t just faster but also scales more easily, reducing the load on your CKAN APIs and databases.

In this post, we’ll explore how PortalJS transforms the frontend experience for data portals, compare it to CKAN’s traditional frontend, and dive into why features like ISR and SSG make a significant difference in performance and scalability.

Leveraging NextJS for Better Performance

When it comes to performance, PortalJS, powered by NextJS, has a significant advantage. One of the key features NextJS brings to the table is its Incremental Static Regeneration (ISR) strategy. ISR allows for the generation of static pages on-demand, which means pages are pre-rendered once and then updated in the background, keeping the load on APIs and databases low.

This is a game-changer for data portals, especially those powered by CKAN, where the Flask-based frontend often regenerates pages dynamically. ISR, in contrast, helps keep pages static until necessary updates are made, making the whole system much more efficient. You get the best of both worlds: the speed of static pages and the freshness of dynamic content.

But NextJS also offers other approaches like Static Site Generation (SSG) and Server-Side Rendering (SSR), which can be applied based on the specific needs of your data portal.

SSG for Static Content

A perfect example of where SSG shines in CKAN is on its homepage. Most of the content on the homepage remains static—the structure, layout, and majority of the information doesn’t change often. While some sections, such as the “latest datasets” or statistics about the number of datasets, organizations, and groups, might be dynamic, these can be handled separately (for example, by re-fetching the dynamic parts on the client side).

By building the CKAN homepage using SSG, you can ensure the page loads super fast, since it’s served as pre-generated static HTML, reducing load on the API and databases.

SSG for Dataset Metadata

Another excellent use case for SSG is dataset metadata pages. In many data portals, dataset metadata—such as the title, description, tags, authors, and sources—are generally created once and remain unchanged. This makes it a great candidate for SSG. Pre-building these pages means they’ll load instantly, without the need for the backend to dynamically generate them with each request. This results in faster page loads and an overall smoother user experience.

By using a combination of these strategies, PortalJS can dramatically reduce latency, improve SEO, and ensure that your data portal is scalable without overwhelming CKAN's APIs or databases.

Improved User Experience

User experience is at the heart of any successful web application, and data portals are no exception. The way users interact with data—whether they’re browsing datasets, filtering through complex metadata, or simply exploring the portal’s interface—directly affects how useful and engaging the platform becomes.

Faster Page Loads

One of the biggest frustrations users face with data portals is slow loading times, especially when the portal is packed with large datasets. CKAN’s Flask-based frontend typically relies on dynamic page generation, which can slow things down as each request hits the server and database.

PortalJS, by contrast, can take advantage of Static Site Generation (SSG) and Incremental Static Regeneration (ISR) to pre-build pages and serve them instantly. The result? Faster load times, snappier navigation, and a much smoother experience for users.

Modern UI/UX Design

With PortalJS, the frontend is decoupled from the backend, giving developers the freedom to craft highly customized user interfaces. You’re no longer restricted by the more rigid CKAN frontend, which can sometimes feel difficult to modify without diving into Python application, Docker set up and dealing with Apache Solr, Postgresql etc.

PortalJS allows developers to integrate modern UI libraries, such as React components, Tailwind CSS, or Material UI, providing a much more polished, user-friendly design. This flexibility lets you create intuitive navigation, visually appealing layouts, and even responsive features that make the portal easier to use across devices.

Seamless Client-Side Interactions

Another major benefit of PortalJS is its ability to handle client-side interactions more effectively. Using dynamic rendering capabilities, features like filtering datasets, sorting results, and searching for content can all be done without constantly reloading the page or hitting the backend server.

This not only improves performance but also creates a more seamless user experience. With CKAN’s built-in frontend, interactions like these often require a round trip to the server, resulting in slower response times and more friction for users.

Scalability & Modularity

Scalability is a critical factor for data portals, especially when handling large datasets and growing user bases. With PortalJS, scaling your application becomes far more straightforward and cost-efficient compared to CKAN’s traditional monolithic frontend, which relies heavily on server-side resources.

Scaling Without a Server

One of the most significant advantages of PortalJS is its ability to serve most of your pages statically, without requiring a dedicated server. Thanks to Static Site Generation (SSG), many pages can be pre-built and then served from a simple object storage solution, such as Cloudflare R2 or AWS S3. This not only reduces infrastructure costs but also dramatically improves the app’s ability to scale since you’re offloading traffic to globally distributed storage networks (CDNs), which are optimized for delivering static assets.

For dynamic content that still requires server-side logic, serverless functions come into play. Pages that need to be generated on the fly can be handled by, for example, AWS Lambda functions, eliminating the need for traditional server infrastructure. By leveraging serverless architecture, you only pay for what you use, and scaling becomes seamless since cloud providers automatically manage resource allocation.

Common Deployment Platforms

PortalJS applications are often deployed using modern hosting platforms like Vercel and Netlify, which make it incredibly easy to scale without the complexities of managing servers. These platforms are built to handle dynamic content using Incremental Static Regeneration (ISR) and support serverless functions to process on-demand requests. Additionally, Cloudflare has started supporting server-side rendering for PortalJS apps, giving developers even more options to balance static and dynamic content delivery.

Scaling on Vercel vs Hosting CKAN on AWS/GCP

Now, let’s compare what it means to scale an app on Vercel versus hosting CKAN’s Python-based Flask application on traditional cloud services like AWS or GCP or Azure. When deploying a CKAN instance, you’re dealing with a monolithic application that requires dedicated servers or virtual machines. You must manage uptime, load balancing, and scaling on your own. If traffic spikes, the infrastructure needs to scale vertically or horizontally, often leading to increased complexity and higher costs.

With Vercel (or platforms like Netlify or Cloudflare Pages), PortalJS handles scaling automatically. Static pages can be served globally via CDNs, and serverless functions take care of dynamic requests. You don’t have to worry about managing backend infrastructure—scaling is built into the platform, and your app adapts to changes in traffic without additional manual configuration.

Developer Experience

A great developer experience (DX) is essential for productivity, especially in fast-paced environments where engineers need to iterate quickly. When it comes to building data portals, PortalJS dramatically simplifies the development workflow compared to CKAN’s traditional frontend setup.

Modern Development Stack

Frontend engineers typically prefer working with a modern JavaScript stack, which includes tools like React, NextJS and TailwindCSS (or other modern CSS frameworks). These technologies are widely adopted, well-documented, and come with robust community support. However, developing with CKAN’s Flask-based frontend can feel much more cumbersome.

To start working on CKAN’s default frontend, developers need to spin up Docker containers with services like PostgreSQL, Apache Solr, Redis, and CKAN (Python app) itself. This setup can take up a lot of storage on the developer's machine, especially when you need multiple CKAN instances for different projects. All these moving parts not only slow down the setup process but also increase the chances of running into configuration issues.

With PortalJS, the workflow is refreshingly simple. Frontend engineers can clone a project and start development with just two commands:

npm install
npm start

That’s it. In a matter of minutes, developers have a fully functional local environment running, without the need to worry about databases, search engines, or caching systems.

Efficient Development with Hot Reloading

Another significant advantage of PortalJS is its support for hot reloading. As soon as you make changes to your code, the browser automatically refreshes to reflect those updates, creating an instant feedback loop. This drastically improves productivity, allowing developers to see their changes in real-time without having to restart services or manually refresh the page.

CKAN does have hot reloading through its Flask development server, which reloads the application when changes are detected. However, the experience may not be as smooth or fast as the hot reloading tools available in modern JavaScript frameworks like NextJS and React. Flask’s hot reloading works well for small changes but might slow down significantly in larger applications, especially if you’re dealing with Docker setups and various integrated services (like PostgreSQL, Solr, and Redis).

Simplified Multi-Project Development

When handling multiple CKAN projects, spinning up individual instances for each one can be a heavy burden on your machine. Each CKAN instance requires its own PostgreSQL database, Solr service, and sometimes even Redis, making it a storage and resource hog. In contrast, PortalJS decouples the frontend entirely from CKAN’s backend. You can easily create and switch between different projects with minimal overhead, saving both time and disk space.

By offloading backend complexity and streamlining the frontend, PortalJS enables a smoother, faster, and more enjoyable development experience for engineers, helping them focus on building beautiful interfaces instead of wrestling with infrastructure.

Conclusion

In the evolving landscape of data portals, delivering an exceptional user experience while ensuring scalability and ease of development is crucial. The PortalJS framework, built on the powerful NextJS, offers a modern solution that addresses the limitations of CKAN's traditional frontend.

Ultimately, choosing PortalJS as a decoupled frontend engine for data portals allows organizations to harness the best of both worlds—robust, high-performing applications that are easy to develop, scale, and maintain. By embracing this innovative approach, data portals can not only meet the demands of today but also adapt to the challenges of tomorrow.