Forem: Parseable

Explain Parquet like I'm Five

Nitish Tiwari — Mon, 12 Aug 2024 10:11:18 +0000

Around early 2010's, as HDFS was at its peak, big data was still growing. It was evident that just the storage system is not enough to handle the data deluge. Innovations were needed to make data processing faster and efficient at the storage format itself.

Doug Cutting, the creator of Hadoop, started Trevni columnar storage format initially. Twitter and Cloudera then collaborated to improve this format. This resulted in the creation of Parquet format. First released in 2013, Parquet is a columnar storage file format that is optimized for large-scale data processing and analytics. Apache Parquet is a top level Apache Software Foundation project since 2015.

In this post we will explore the structure of Parquet files, its advantages, and use cases. We will also discuss best practices for working with Parquet files. We'll also look at why we at Parseable chose Parquet as our primary storage format and the challenges we face.

What is a columnar format?

A column oriented format differs structurally from row oriented formats such as CSV or Avro. The easiest way to understand this is to start with a simple dataset in a row oriented file such as a CSV, like this:

ID	First Name	Last Name	City	Age
1	Alice	Brown	Boston	29
2	Bob	Smith	New York	35
3	Charlie	Davis	San Francisco	42

When this data is stored in a CSV file, it is stored row-wise, i.e. data from a row is next to data from the next row and so on. Each row contains all the columns for a single record. So the actual physical storage of the data would look like this:

1,Alice,Brown,Boston,29;2,Bob,Smith,New York,35;3,Charlie,Davis,San Francisco,42;

In contrast, a columnar file stores data from a column together. So, the same data in a columnar file would be physically stored like this:

1,2,3;Alice,Bob,Charlie;Brown,Smith,Davis;Boston,New York,San Francisco;29,35,42;

Sure, it is difficult to read for humans, but this format is much more efficient for certain types of queries. For example, if you wanted to know the average age of all the people in the dataset, you would only need to read the Age column. In a row-based file, you would have to read the entire file to get the same information.

Implications of columnar format

The way data is organized in in a columnar format has several important implications (both good and bad) on how data is stored, read, and processed. Let's examine some of these implications:

Writing the file: While writing a columnar file, the writer needs to know the all the values of a column for all the rows (that need to be written). This means that the writer needs to buffer the entire column in memory before writing it to disk. This can be a challenge for very large datasets. Parquet and other columnar formats added a way around, that we'll review later.
Column level configuration: A columnar file stores data from one column together physically, it can use different compression and encoding schemes for each column. For example, if there is a column with repeating values, RLE or Dictionary encoding will be more efficient than other encoding schemes. Such flexibility allows optimizing storage and query performance for each column.
Random access: Because of relatively complex format, data in a columnar file is difficult to be randomly accessed. For example, if you want to read a row from a columnar file, you would have to read the footer, then the metadata, then the column chunks, and finally the pages. You may even be able to skip a few steps, but you'll need to read a whole page at the least to get the data you need. This is in contrast to row-based files where you can read a row directly.
Schema evolution: Schema evolution is the process of changing the schema of a dataset. In a columnar file, schema evolution is next to impossible. This is because the schema is stored in the metadata and changing the schema means changing the metadata. This can be a challenge when working with large datasets.
Interaction with files: Columnar files are not human-readable. This means that you can't open a columnar file in a text editor and read the data. You need a special tool or library to read the data. This can be a challenge when working with small datasets or when debugging. Since, each columnar file format is different in the structure, each of then need their own reader and writer implementations in different languages, only then end users can interact with the files.

Parquet file format

At a high level, a Parquet file is a collection of row groups, each of which contains column chunks. Each column chunk is further divided into pages. The file also contains metadata that helps in reading and decoding the data efficiently.

Let's take a visual example. In below example, there are N columns in this table, split into M row groups. The file metadata contains the locations of all the column chunk start locations.

4-byte magic number "PAR1"

  <Row Group 1>
    <Column 1 Chunk 1>
    <Column 2 Chunk 1>
    ...
    <Column N Chunk 1>
  </Row Group 1>

  <Row Group 2>
    <Column 1 Chunk 2>
    <Column 2 Chunk 2>
    ...
    <Column N Chunk 2>
  </Row Group 2>

  <Row Group M>
    <Column 1 Chunk M>
    <Column 2 Chunk M>
    ...
    <Column N Chunk M>
  </Row Group M>

File Metadata
4-byte length in bytes of file metadata (little endian)
4-byte magic number "PAR1"

Row Groups

A writer needs to know all the values of a column before writing it to disk. Why? let's understand this better, with an example. Take our previous table.

ID	First Name	Last Name	City	Age
1	Alice	Brown	Boston	29
2	Bob	Smith	New York	35
3	Charlie	Davis	San Francisco	42

If you were to write this data in a columnar file, you would need to write all the ID values together, then all the First Name values together, and so on. The final file looks like:

1,2,3;Alice,Bob,Charlie;Brown,Smith,Davis;Boston,New York,San Francisco;29,35,42;

Now, imagine if this file was not complete, yet. Say, there are 5 more rows to be appended to the file (after first 3 rows are written in the Parquet file).

One way to do this is, read the entire file, append the new rows, and write the entire file back to disk. This would need a lot of data shuffling to keep the columnar format intact. Other way would be to instead keep all the rows in memory (till you get all the 8 rows) and write them to disk, and don't allow appends (once the file is written).

Now think of production workloads where writers need to write millions of rows worth of data to a Parquet file. Neither the shuffling of data nor the loading of all rows in memory is efficient. To solve this problem, Parquet introduced the concept of row groups.

Row groups are essentially a way to divide the data in smaller chunks, so they can be efficiently written to disk. Each row group contains a subset of the rows and all the columns for those rows. The writer needs to load only the rows in a row group to write them to disk, and then wait for next set of rows. When the file is complete, the writer can write the metadata to the file.

Column Chunks

Within a Row Group, data is stored by columns. These column chunks are the core of Parquet's columnar storage format. By storing data this way, Parquet can efficiently read only the columns needed for a query.

Pages: Each column chunk is further divided into pages. Pages are the smallest unit of data storage in a Parquet file. They contain the actual data and metadata necessary for decoding.
Metadata: Parquet files contain metadata at different levels. File-level metadata includes information about the schema and the number of row groups. Row group metadata provides details about column chunks such as their size and encoding. Column chunk metadata includes information about pages. Other page header metadata is used to read and decode the given data.

Supported Types

The data types supported by a Parquet file are minimal to help reduce the complexity of implementing readers and writers for the file type. Further, the user can use a logical type to extend the types stored in your file. These annotations are stored in the metadata file as LogicalTypes.md.

Some of the supported data types include:

BOOLEAN: 1 bit boolean
INT32: 32-bit signed ints
INT64: 64-bit signed ints
INT96: 96-bit signed ints
FLOAT: IEEE 32-bit floating point values
DOUBLE: IEEE 64-bit floating point values
BYTE_ARRAY: arbitrarily long byte arrays
FIXED_LEN_BYTE_ARRAY: fixed length byte arrays

Building and Working with Parquet

Working with Parquet files in different programming environments is straightforward thanks to the wide range of libraries and tools available. For example, if you want to work with Oracle Databases, you can opt for Python-based parquet tools.

For example, parquet-tools in Python is a very popular CLI tool to interact with Parquet files.

pip3 install parquet-tools

Refer parquet-tools docs here.

Use Cases

Parquet is particularly advantageous in scenarios that involve large-scale data processing and analytics.

Big Data Processing with Apache Spark: Apache Spark comes with Parquet as the default storage format. Its compression and encoding schemes enable Spark to efficiently read and write large datasets, run fast analytical queries, and save storage space.
Data Warehousing: Parquet's efficient storage and quick query performance in data warehousing are excellent for storing and analyzing large volumes of data. Data warehousing solutions such as Amazon Redshift and Google BigQuery are common use cases.
Machine Learning: Parquet files are the file format of choice for machine learning applications due to their efficient storage and retrieval of large datasets. Data scientists use fast data loading and processing to accelerate model training and evaluation.

Best Practices

To optimize your work with Parquet files, consider the following best practices:

Column Pruning: When querying data, select only the columns you need. This reduces the amount of data read from disk and improves query performance.
Predicate Pushdown: Use predicates in your queries to filter data early in the read process. Parquet supports predicate pushdown, which helps minimize the amount of data read.
Compression and Encoding: Based on your data characteristics, choose the proper compression and encoding schemes. Experiment with different options to find the best balance between storage space and query performance.
Batch Processing: Process Parquet files in batches to take advantage of Parquet's efficient row group and page structures. This improves I/O performance and reduces overhead.

What is Syslog? Components, benefits, and best practices

Jen Wike Huger — Tue, 30 Jul 2024 13:33:59 +0000

The team at Parseable is writing a ton of content around how to do logging in a lightweight and powerful way. One of their recent installments is about Syslog. Read their blog to learn more.

Syslog is the abbreviation for System Logging Protocol. It is a logging method that allows monitoring and managing logs from different parts of the server or system. The primary use of Syslog data is to trace back errors in case of a failure or event. It can also be used to get information about your system's overall performance.

This Syslog guide will help you understand how it works and its functionality. Let's begin by understanding what it is and how it evolved.

History and Evolution of Syslog

The journey of Syslog began in the early days of the Unix operating system where it served as a simple logging protocol for capturing system messages. Eric Allman created the first instance of Syslog in the late 1980s. Soon it became a must-have tool for Unix administrators. The growing need for a standard logging protocol eventually led to the publication of RFC 3164 the first Syslog standard. It has been further developed and extended into RFC 5424 to establish a more solid and in the meantime flexible basis for the frameworks of message formats that could be used with Syslog.

3 Core Components of Syslog

At its core Syslog consists of several key components and concepts:

Syslog Daemon: The Syslog daemon is responsible for receiving processing and forwarding Syslog messages. Some examples of Daemons include Syslog rsyslog or syslog-ng. It is the central hub for logging activities. Hence it helps capture and appropriately handle all system events along with log data.

Syslog Messages: A typical Syslog message is made up of three parts:

The priority (PRI)
Header
Message (MSG)
The PRI combines facility code and severity indicating the message's source and importance. The header includes a timestamp and the hostname or IP address while the MSG contains the log message or event data.

Facilities and Severities: Syslog categorizes messages by facilities such as kern (kernel messages) user (user-level messages) and mail (mail system). Severities range from emergency (system unusable) to debug (debug messages) allowing administrators to filter and prioritize log data effectively.

How Syslog Works

Syslog operates through a straightforward yet powerful workflow:

Message Flow: Applications generate Syslog messages which are transported over networks using protocols like UDP (User Datagram Protocol) or TCP (Transmission Control Protocol). Upon arrival the Syslog or log server processes and stores these messages according to configured rules.

Syslog Formats: The most common standard Syslog message formats include RFC 3164 and RFC 5424. RFC 3164 is the original Syslog format and is widely supported. The RFC 5424 offers enhanced features including structured data and better timestamp precision.

Benefits of Using Syslog

Several key benefits drive Syslog's widespread adoption.

Centralized Logging: By aggregating logs from multiple sources into a central location Syslog simplifies monitoring and management. This centralized approach enhances visibility and enables more efficient analysis of log events.

Real-time Monitoring: Syslog supports real-time alerting and Syslog monitoring allowing administrators to detect and respond to issues promptly. This capability is crucial for maintaining system uptime and security across network devices.

Compliance and Auditing: Syslog helps organizations meet regulatory requirements by providing a reliable and tamper-evident logging mechanism. It facilitates auditing and records all critical events in the log file.

Troubleshooting: Syslog's detailed and structured logs are invaluable for troubleshooting. By examining log messages administrators can more efficiently identify and resolve issues minimizing downtime and disruption.

Syslog Configuration and Usage
Configuring Syslog involves setting up the Syslog daemon and integrating it with various Syslog tools:

Configuring Syslog Daemon
Basic configuration typically involves specifying log file locations setting log rotation policies and defining message filtering rules. Advanced configurations can include:

Forwarding logs to a central Syslog server
Applying TLS/SSL for secure transmission
Customizing log formats
Common Syslog Tools
Syslog provides the ability to use tools to enhance performance. One of the most famous tools is RSyslog. It is known for its high performance and flexibility and as a result it is widely preferred in modern Linux environments. Syslog-ng offers advanced features like content-based filtering and high-availability clustering. Moreover if you integrate Syslog with log management systems such as Parseable it provides powerful analytics and visualization capabilities.

Best Practices for Syslog

To maximize Syslog's effectiveness administrators should follow these best practices:

Security
Ensure secure transmission of Syslog messages using TLS/SSL. This prevents unauthorized access and protects the integrity of log data.

Performance
Optimize Syslog performance by fine-tuning buffer sizes configuring appropriate log rotation policies and using efficient storage solutions.

Scalability
Plan for scalability by distributing logging workloads across multiple servers and using load-balancing techniques. This ensures that Syslog can handle large volumes of logs in high-demand environments.

Maintenance
Regular maintenance tasks include:

Monitoring disk usage
Checking for log file corruption
Updating Syslog configurations as needed
This proactive approach helps maintain a healthy logging infrastructure.

Challenges and Limitations of Syslog

Despite its benefits Syslog faces several challenges:

Reliability Issues: UDP-based transport while fast can be unreliable leading to potential message loss. Using TCP or implementing reliable delivery mechanisms can mitigate this risk.

Scalability Concerns: Handling large volumes of logs requires careful planning and resource allocation. Implementing distributed logging solutions and efficient indexing can help address scalability challenges.

Complexity: Managing complex Syslog configurations can be daunting. Clear documentation regular audits and leveraging automation tools can simplify configuration management.

Future of Syslog

Syslog continues to evolve adapting to modern IT environments:

Emerging Trends: New developments in logging technologies such as structured logging and enhanced metadata support are enhancing Syslog's capabilities and driving improvements in log analysis and correlation.

Integration with Modern Systems: Syslog is increasingly integrating with cloud-native and containerized environments. This includes support for logging drivers in Docker and Kubernetes enabling seamless logging across diverse infrastructures.

Conclusion

Syslog remains a fundamental tool for system logging offering numerous benefits for IT professionals. Administrators can leverage Syslog to enhance system monitoring compliance and troubleshooting by understanding its history core concepts and best practices. Syslog adapts as the logging landscape evolves ensuring its relevance in modern computing environments.

How to set up a CDC pipeline to capture and analyze real-time database changes with Parseable

Jen Wike Huger — Mon, 22 Jul 2024 16:49:11 +0000

Databases are critical for any application. Data constantly gets updated, inserted, and deleted. In most of cases it is important for the business to keep track of these changes due to security concerns, auditing requirements, and to keep other relevant systems up to date.

Change Data Capture (CDC) has gained popularity, precisely to address this problem. CDC is a technique used to track all changes in a database and capture them in destination systems. Debezium is a popular CDC tool that leverages database logs as the source of truth, and streams the changes to Kafka and compatible systems like Redpanda.

The key thing to note here is that while databases are typically mutable, the changes captured by the CDC system should be immutable. Only then users have a system of record that can be trusted. Parseable is built for immutable logs and streams of events and data. It is a great fit for CDC targets. The way we designed Parseable allows for long term retention and querying of logs, and it also provides a rich set of tools for analysis and monitoring.

In this tutorial, we use Redpanda, PostgreSQL, and Debezium to ingest CDC events into Parseable.

Metadata to actionable insights in Grafana: How to view Parseable metrics

Jen Wike Huger — Thu, 11 Jul 2024 15:55:59 +0000

Parseable deployments in the wild are handling larger and larger volumes of logs, so we needed a way to enable users to monitor their Parseable instances.

Typically this would mean setting up Prometheus to capture Parseable ingest and query node metrics and visualize those metrics on a Grafana dashboard. We added Prometheus metrics support in Parseable to enable this use case.

But we wanted a simpler, self-contained approach that allows users to monitor their Parseable instances without needing to set up Prometheus.

This led us to figuring out a way to store Parseable server's internal metrics in a special log stream called pmeta.

This stream keeps track of important information about all of the ingestors in the cluster. This includes information like the URL of the ingestor, commit id of that ingestor, number of events processed by the ingestor, and staging file location and size.

This is a sample event in the pmeta stream.

{
  "address": "http://ec2-3-136-154-35.us-east-2.compute.amazonaws.com:443/",
  "cache": "Disabled",
  "commit": "d6116e8",
  "event_time": "2024-07-02T09:49:05.125255417",
  "event_type": "cluster-metrics",
  "p_metadata": "",
  "p_tags": "",
  "p_timestamp": "2024-07-02T09:49:05.540",
  "parseable_deleted_events_ingested": 35095373,
  "parseable_deleted_events_ingested_size": 10742847195,
  "parseable_deleted_storage_size_data": 1549123461,
  "parseable_deleted_storage_size_staging": 0,
  "parseable_events_ingested": 3350101,
  "parseable_events_ingested_size": 1054739567,
  "parseable_lifetime_events_ingested": 38445474,
  "parseable_lifetime_events_ingested_size": 11797586762,
  "parseable_lifetime_storage_size_data": 1732950386,
  "parseable_lifetime_storage_size_staging": 0,
  "parseable_staging_files": 2,
  "parseable_storage_size_data": 183826925,
  "parseable_storage_size_staging": 0,
  "process_resident_memory_bytes": 113250304,
  "staging": "/home/ubuntu/parseable/staging"
}

Let's show you how to visualize this data in a Grafana dashboard.

We'll start by setting up Parseable to collect this pmeta data.

Getting Started with Parseable

Parseable is a cloud-native log management solution that efficiently handles large-scale log data. By integrating Parseable with your infrastructure, you can streamline log ingestion, storage, and querying, making it an essential tool for observability and monitoring.

You can choose the right installation process for you.

To quickly install Parseable using Docker, open the terminal and type the command:

docker run -p  8000:8000 \  
-v /tmp/parseable/data:/parseable/data \  
-v /tmp/parseable/staging:/parseable/staging \  
-e  P_FS_DIR=/parseable/data \  
-e  P_STAGING_DIR=/parseable/staging \  
containers.parseable.com/parseable/parseable:latest \  
parseable local-store

You can verify the installation by accessing the Parseable UI by navigating to http://localhost:9000 in your web browser. Log in using the default credentials (admin/admin) and explore the dashboard to ensure everything is set up correctly.

Finally, we need to create a log stream before we can send events. A log stream is like a project that will essentially store all your ingested logs. For this tutorial, we'll have a log stream named pmeta. To create a log stream, log in to your Parseable instance, and you'll find a button on the right-hand top side.

Note that:

pmeta is automatically created and populated in a Parseable cluster (high availability setup)
pmeta is not created in a single node setup
if you're not interested in this data, you can set the retention to 1 day for the pmeta stream to avoid storing this data

Read more about the pmeta stream in the Parseable documentation.

Instal Grafana and the Parseable plugin

Grafana helps you collect, correlate, and visualize data with beautiful dashboards. We'll connect the parseable instance with Grafana via the Parseable Grafana datasource. This plugin allows you to query Parseable data using SQL and visualize it in Grafana.

If you want to self-host Grafana, you can host it on a dedicated cloud instance or locally, depending on your requirements. Follow the official Grafana installation guide for more information.

Once the Grafana instance is setup, let's quickly install the Parseable plugin and connect our Parseable instance to Grafana.

Login to your Grafana instance and navigate to the administration setting on the left-hand side menu.
Click on Plugin and Data Option

Open the Plugins page and search for Parseable

Install the plugin and then click on Add New Datasource

From the datasource page, fill in the following details:

In the URL field, type the URL of your Parseable query instance. For example, https://demo.parseable.com:443.

Under the Auth Section, switch to the Basic Auth setting.

In the Basic Auth Details section, enter your Parseable username and password.

Finally, click on Save & Test to verify the connection.

Setting up the Grafana dashboard

We'll now use the Parseable Data Source to query data from the Parseable Meta Stream (pmeta).

Navigate to the Dashboard section and click on New and select Import Dashboard.

Enter the Dashboard ID as 21472 and click on load.

Ensure the data source to Parseable-DataSource by selecting from the dropdown menu.

Once done, click on import. It should take a few seconds to load, and then it will create the dashboard.

We query data from Parseable using SQL. To learn more about querying data in Parseable, you can refer to our documentation.

Summary

Now, you've learned how to create a Grafana dashboard using Parseable's pmeta stream.

This dashboard provides crucial insights into your Parseable instance's performance, and we encourage you to customize this dashboard further to fit your specific needs.

🏃🏽‍♀️ To see Parseable in action, watch this YouTube video. Get started with Parseable in just a single command.

💬 For technical questions or to share your implementations, join our developer community on Slack.

📝 Read more from the Parseable blog.

Ready to enhance your observability? Start using Parseable and Grafana today to unlock the full potential of your log data.

Ingesting Data to Parseable Using Pandas

Jen Wike Huger — Wed, 26 Jun 2024 18:40:04 +0000

A Step-by-Step Guide

Managing and deriving insights from vast amounts of historical data is not just a challenge but a necessity. Imagine your team grappling with numerous log files, trying to pinpoint issues. But, because logs are stored as files, it is very inefficient to search through them. This scenario is all too familiar for many developers.

Enter Parseable, a powerful solution to analyze your application logs. By integrating with pandas, the renowned Python library for data analysis, Parseable offers a seamless way to ingest and leverage historical data without the need to discard valuable logs.

In this post, we explore how Parseable can revolutionize your data management strategy, enabling you to unlock actionable insights from both current and archived log data effortlessly.

Requirements

Python installed on your system
Pandas library
Requests library
A CSV file to be ingested
Access to the Parseable API

The CSV File

Our code example is based on a Kaggle Dataset. We've used a CSV file named e-shop_clothing_2008.csv. Feel free to use your own dataset to follow along. First, ensure your CSV file is formatted correctly and accessible from the script's directory.

The Parseable API

Next, we'll interact with the Parseable API to send our data. Here we're using the demo Parseable instance. Before sending any data, please ensure you have entered the correct endpoint and credentials:

Endpoint: https://demo.technocube.in/api/v1/ingest
Username: admin
Password: admin

Writing the Script

Here’s a Python script that reads the CSV file in chunks and sends each chunk to the Parseable API (in a stream called testclickstream). Replace the CSV file path, Parseable endpoint, and authentication credentials with your own.

import pandas as pd
import requests
import json

# Define the CSV file path
csv_file_path = 'e-shop_clothing_2008.csv'

# Define the Parseable API endpoint
parseable_endpoint = 'https://demo.technocube.in/api/v1/ingest'

# Basic authentication credentials
username = 'admin'
password = 'admin'

headers = {
    'Content-Type': 'application/json',
    'X-P-Stream': 'testclickstream'
}

# Read and process the CSV file in chunks
chunk_size = 100  # Number of rows per chunk
for chunk in pd.read_csv(csv_file_path, chunksize=chunk_size, delimiter=';'):
    # Convert the chunk DataFrame to a list of dictionaries
    json_data = chunk.to_dict(orient='records')

    # Convert list of dictionaries to JSON string
    json_str = json.dumps(json_data)

    # Send the JSON data to Parseable
    response = requests.post(parseable_endpoint, auth=(username, password), headers=headers, data=json_str)

    # Check the response
    if response.status_code == 200:
        print('Chunk sent successfully!')
    else:
        print(f'Failed to send chunk. Status code: {response.status_code}')
        print(response.text)

Explanation of the Script

We have divided the code's flow into six steps to help you understand its function. This will also help you understand how Pandas libraries and Parseable work together.

Importing Libraries
The script starts by importing Pandas for data manipulation, Requests for HTTP requests, and JSON for handling JSON data.
Defining File Path and Endpoint: Specify the path to the CSV file and the Parseable API endpoint. Replace these with your actual file path and API endpoint.

Authentication and Headers
Set up basic authentication credentials and headers. The X-P-Stream header indicates the stream or collection name.
Reading CSV in Chunks: Use pd.read_csv to read the CSV file in chunks of 100 rows. The chunk size parameter handles large files efficiently without memory issues.

Converting Data to JSON
Convert each chunk to a list of dictionaries using to_dict with orient='records', then to a JSON string.

Sending Data to Parseable
Send the JSON data to the Parseable API using a POST request. Check the response status code to ensure successful ingestion. Print any errors.

Handling Errors and Retries

Network issues or server errors might prevent successful data ingestion in real-world scenarios. To make the script more robust, implement error handling and retries. Also, look for code errors, if any.

Next Steps

Ingesting data into Parseable using Pandas is straightforward and efficient. By reading data in chunks and converting it to JSON, we can seamlessly send it to the Parseable API.

This script serves as a foundation and is customizable to your specific needs, including sophisticated error handling, logging, or parallel processing.

Follow this guide to integrate Pandas and Parseable effectively, ensuring smooth and efficient data ingestion for your projects.

To get started or try Parseable, visit our demo page.

Build complete website with Docusaurus

Surya Shakti — Sat, 22 Oct 2022 15:40:27 +0000

In this post, we'll look at how to build an end to end product website including landing page(s), docs pages and blogs using Docusaurus.

Installing Docusaurus

For building a Docusaurus website we have to install it first with a starter website skeleton. Run the below command:

npx create-docusaurus@latest my-website classic

After the successful installation you will get the basic website which you can run right away and start customizing it according to you needs.

Seeing website in development server

To run the development server go to the root directory of the project and run the the following command.

npm start

After starting the development server you will can see your site on http://localhost:3000/. It will look something like the following image.

This boiler plate we get after installing the Docusaurus have the following folder structure

my-website
├── blog
│   ├── 2019-05-28-hola.md
│   ├── 2019-05-29-hello-world.md
│   └── 2020-05-30-welcome.md
├── docs
│   ├── doc1.md
│   ├── doc2.md
│   ├── doc3.md
│   └── mdx.md
├── src
│   ├── css
│   │   └── custom.css
│   └── pages
│       ├── styles.module.css
│       └── index.js
├── static
│   └── img
├── docusaurus.config.js
├── package.json
├── README.md
├── sidebars.js
└── yarn.lock

Here we can see different directories which contain different files. Let us discuss about all those things.
On first we have the blog directory. It contains the Markdown files of the blogs you want to show in the website.
The docs directory contains the docs of the website.
In src you have CSS files and your pages directory which have your landing page of the website and here you can add other custom pages to according to your requirements.
In static directory we have our assets like images and company logo.

Customizing Home Page

src/pages/index.js is the homepage for the website. It exports a React component that is rendered between navbar and footer. we will also see how to customise navbar and footer later.

The boiler plate code of the home page is :

import React from 'react';
import clsx from 'clsx';
import Link from '@docusaurus/Link';
import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
import Layout from '@theme/Layout';
import HomepageFeatures from '@site/src/components/HomepageFeatures';

import styles from './index.module.css';

function HomepageHeader() {
  const {siteConfig} = useDocusaurusContext();
  return (
    <header className={clsx('hero hero--primary', styles.heroBanner)}>
      <div className="container">
        <h1 className="hero__title">{siteConfig.title}</h1>
        <p className="hero__subtitle">{siteConfig.tagline}</p>
        <div className={styles.buttons}>
          <Link
            className="button button--secondary button--lg"
            to="/docs/intro">
            Docusaurus Tutorial - 5min ⏱️
          </Link>
        </div>
      </div>
    </header>
  );
}

export default function Home() {
  const {siteConfig} = useDocusaurusContext();
  return (
    <Layout
      title={`Hello from ${siteConfig.title}`}
      description="Description will go into a meta tag in <head />">
      <HomepageHeader />
      <main>
        <HomepageFeatures />
      </main>
    </Layout>
  );
}

you can edit here like you would edit a normal react component and it will change in the website.

Customizing Navbar

In the docusaurus.config.js we have themeConfig object where we can customize our Navbar and Footer.

For navbar customization, we can do it in this object

 navbar: {
        title: 'My Site',
        logo: {
          alt: 'My Site Logo',
          src: 'img/logo.svg',
        },
        items: [
          {
            type: 'doc',
            docId: 'intro',
            position: 'left',
            label: 'Tutorial',
          },
          {to: '/blog', label: 'Blog', position: 'left'},
          {
            href: 'https://github.com/facebook/docusaurus',
            label: 'GitHub',
            position: 'right',
          },
        ],

Here in title you can give title on the website. In logo you can add the company's logo. In items you can have the the links of different pages or different sections.
The key position in item object decides the position of that item in the navbar.

Customizing the footer

footer: {
        style: 'dark',
        links: [
          {
            title: 'Docs',
            items: [
              {
                label: 'Tutorial',
                to: '/docs/intro',
              },
            ],
          },
          {
            title: 'Community',
            items: [
              {
                label: 'Stack Overflow',
                href: 'https://stackoverflow.com/questions/tagged/docusaurus',
              },
              {
                label: 'Discord',
                href: 'https://discordapp.com/invite/docusaurus',
              },
              {
                label: 'Twitter',
                href: 'https://twitter.com/docusaurus',
              },
            ],
          },
          {
            title: 'More',
            items: [
              {
                label: 'Blog',
                to: '/blog',
              },
              {
                label: 'GitHub',
                href: 'https://github.com/facebook/docusaurus',
              },
            ],
          },
        ],
        copyright: `Copyright © ${new Date().getFullYear()} My Project, Inc. Built with Docusaurus.`,
      },

In links object each object is a column in the footer which contains links.

Creating a new page

To create a new page for your website you can add it in src/pages directory.

├── src
│   ├── css
│   │   └── custom.css
│   └── pages
│       ├── styles.module.css
│       └── index.js
│       └── new-page.js

Here as we added new-page.js it became a route itself.
you can navigate to /new-page in the browser to see.

Conclusion

In this post we saw how to setup a custom website with Docusaurus and also learnt how to customize the navbar, footer etc.

Logging for your Node.js app

Surya Shakti — Thu, 22 Sep 2022 07:03:31 +0000

Introduction

In this article we will learn how we can store logs of Node.js application into parseable using Winston library. We'll look at what is logging and why it is important.

What is logging?
Using winston in a Node.js application
sending those logs to the parseable

What is logging?

Logging is the process of recording application events and data to a log file or any other sources, for the purpose of analyzing the system.

Logs help developers to find the error, track the source of the bug and fix it. In Node.js it is critical to structure the application logs. when we are at the development phase we can use console.log to find the problems and to get the information you need.

But once the application is in production we cannot use the console.log any more.

Using winston in a Node.js application

I assume you have a node.js application, now to install winston & winston-transport in your project run the following commands.

npm install winston
npm install winston-transport

Now our aim is to replace all the console messages from the application with the winston logger. So Just for example if you have these three event logs in your node.js application which you want to store in parseable using winston library.

connsole.warn('Got mutiple elements with same id');
console.error('Login Failed. Invalid ID');
console.info('Events posted successfully.');

Configuring Winston in Node.js app

Create a 'logger' folder in the root directory where we will configure winston and send the logs to parseable.
Now in logger/index.js add following code

const winston = require("winston");
const { combine, timestamp, label, printf } = winston.format;
const Transport = require("winston-transport");
var axios = require("axios");

const myFormat = printf(({ level, message, label, timestamp }) => {
  return JSON.stringify({
    timestamp: timestamp,
    level: level,
    message: message,
  });
});

class CustomTransport extends Transport {
  constructor(opts) {
    super(opts);
  }
  log(info, callback) {
    console.info(info);
    var data = JSON.stringify([info]);

    var config = {
      method: "post",
      url: `https://demo.parseable.io/api/v1/logstream/${streamName}`,
      headers: {
        "X-P-META-Tag": "Owner",
        "X-P-META-Tag": "Host",
        "X-P-META-Tag": "Host.owner",
        Authorization: `Basic ${key}`,
        "Content-Type": "application/json",
      },
      data: data,
    };

    axios(config)
      .then(function (response) {
        console.log(response.data);
      })
      .catch(function (error) {
        console.log(error);
      });

    callback();
  }
}

const devLogger = () => {
  const transport = new CustomTransport({});

  return winston.createLogger({
    level: "debug",
    format: combine(label(), timestamp(), myFormat),
    transports: [transport],
  });
};


let logger = null;

if (process.env.NODE_ENV !== "production") {
  logger = devLogger()  
}

module.exports = logger;

Here we are configuring winston in our node.js application and sending the logs to parseable.

Let's discuss this code in parts for better understanding.

Initializing winston logger instance

const devLogger = () => {
  const transport = new CustomTransport({});
  return winston.createLogger({
    level: "debug",
    format: combine(label(), timestamp(), myFormat),
    transports: [transport],
  });
};

The snippet above contains the initialization of a Winston logger instance. Here we specify the log level for this specific logger instance using the npm log level standard, format in which logs will be stored and


 that specifies where the logs data will go. In our case we will send it to the parseable.



> **Setting custom format of the log data**
>
>>

 ```js 
const myFormat = printf(({ level, message, label, timestamp }) => {
  return JSON.stringify({
    timestamp: timestamp,
    level: level,
    message: message,
  });
});

The snippet above specifies the format of the log data in which it will be stored.

Sending the log data to parseable

class CustomTransport extends Transport {
  constructor(opts) {
    super(opts);
  }
  log(info, callback) {
    console.info(info);
    var data = JSON.stringify([info]);
    var config = {
      method: "post",
      url: `https://demo.parseable.io/api/v1/logstream/${streamName}`,
      headers: {
        "X-P-META-Tag": "Owner",
        "X-P-META-Tag": "Host",
        "X-P-META-Tag": "Host.owner",
        Authorization: `Basic ${key}`,
        "Content-Type": "application/json",
      },
      data: data,
    };
    axios(config)
      .then(function (response) {
        console.log(response.data);
      })
      .catch(function (error) {
        console.log(error);
      });
    callback();
  }
}

The snippet above is responsible for sending the log data to the parseable. here streamName is the log stream you have created in parseable. and the key is the authentication key for accessing the parseable.

calling the logger function

let logger = null;
if (process.env.NODE_ENV !== "production") {
  logger = devLogger()  
}

You may not want to store the logs in parseable in development phase then you can see them on console to test it and when in production mode you can send it to parseable. The snippet above calls the logger function according to your requirements. you can also call the function directly if you don't want this conditional calling.

Then we can use logger instead of console in our application.

const logger = require('./logger')

logger.warn('Got mutiple elements with same id')
logger.error('Login Failed. Invalid ID')
logger.info('Events posted successfully.')

Conclusion

Hurray!!🥳🥳🥳

You have successfully integrated parseable with your node.js application. now you can run your application and all the events you have replaced with logger will be posted to parseable and you can check in the logstream you created earlier.

Reimagine log storage: Parseable

Nitish Tiwari — Tue, 20 Sep 2022 10:22:49 +0000

Context

Whether you're a Developer or SRE or DevOps - when you're tasked with setting up logging, there are essentially two options:

Search Engines

Setup an indexing based search engine, masquerading as log storage. Such products are difficult to deploy and run in longer term - manage indexes, local and remote storage, different node types and so on.

Additionally, indexing in the ingestion flow causes high CPU and Memory consumption, while denying very high ingestion rates.

SaaS Platforms

Alternatively, pay for an exorbitantly costly SaaS platform that is very easy to get started with. As time grows, data volumes and costs increase.

But data gravity means getting out of this platform is difficult and you end up storing a fraction of log data to save costs.

We dealt with both these options in our work and we know both these options are not ideal. This pushed us to think what would a modern, cloud native, log storage and observability platform look like.

It is clear that log storage of future won't be another index based search engine in a new language.

We set out to build a completely indexing free log storage platform. This led to Parseable.

Introduction

Parseable is a simple, efficient and fast log storage and observability platform. Think the simplicity of Prometheus but for Logs. Written in Rust, Parseable leverages Apache Arrow, Parquet and widely available object storage platforms for efficiency, cost effectiveness and performance.

It is compatible with standard logging agents like FluentBit, LogStash etc. Parseable also offers a builtin, intuitive GUI for log query and analysis.

Conclusion

As we launched Parseable, we see tremendous interest in the community to try this out. We'd love for you to try out Parseable and we're all ears for any questions, feedback, and comments.

Get Started: https://www.parseable.io/docs/quick-start
Slack: https://launchpass.com/parseable
Github: https://github.com/parseablehq/parseable
Documentation: https://www.parseable.io/docs/introduction

Logging for your frontend apps

Surya Shakti — Mon, 19 Sep 2022 11:46:24 +0000

Logging is one of the key activities when you set out to build a reliable piece of software. Yet, we've seen developers putting off logging setup for future. This can be largely attributed to unavailability of simple, developer friendly logging platforms that just work.

In this post we'll take a look at how to configure a React app to send logs (over HTTP) to Parseable - an open source, developer friendly log storage and query platform.

The Setup

While Parseable is FOSS and can be setup anywhere, we'll use the public demo instance to avoid the setup and instead focus on the react app side of things.

I'll use the Pino JS Logger as the logging client.

Create a React app

Ideally you may have a react app available that you want to add logging for. In case you're starting from scratch, create a react application first so that we can add logging to the app.

npx create-react-app my-project-demo

Setup Pino

Install Pino so we can use it as a library in the react app.

npm install pino

Once installed successfully, create a directory named src/loggers inside your react app source. We'll initialise Pino library here, to send logs to Parseable.

Create a file src/loggers/index.js and paste this following code.

import pino from "pino";

const send = async function (level, logEvent, a, b) {
  const url = "https://demo.parseable.io/api/v1/logstream/pinotest";

  const response = await fetch(url, {
    method: "POST",
    headers: {
      Authorization: "Basic cGFyc2VhYmxlOnBhcnNlYWJsZQ==",
      "Content-Type": "application/json",
    },
    body: JSON.stringify([logEvent]),
  });
  console.log(response);
};

const logger = pino({
  browser: {
    serialize: true,
    asObject: true,
    transmit: {
      send,
    },
  },
});

export default logger;

Note that the Parseable URL https://demo.parseable.io/api/v1/logstream/pinotest points to the pinotest stream on Demo Parseable instance. This is a public instance to demonstrate Parseable UI. Please don't post any sensitive / production logs to the URL. Refer to Parseable Docs to start your own Parseable instance.

Add logging to the app

Now we can add logs to the react app we created initially.

import logger from "./loggers";
logger.info("test log! pinotest stream from reactjs application.");
function App() {
  return <div className="App">ReactJs logs to parseable</div>;
}
export default App;

That is all! You have successfully integrated parseable with your React app.

All the events you log with the logger will be posted to Parseable and you can check in the log stream you created earlier.

Conclusion

In this post we learnt about how to setup and configure a react app to send logs to Parseable - a simple, developer friendly log storage platform. We were able to do this with just few lines of code.

With this setup, all the logs are stored securely and efficiently on a remote machine. Logs are available in seconds for auditing and debugging purposes.

Links:
[1] Parseable Github: https://github.com/parseablehq/parseable
[2] Parseable Docs: https://www.parseable.io/docs/introduction
[3] Pino Github: https://github.com/pinojs/pino
[4] Pino Docs: https://getpino.io/

Forem: Parseable

Explain Parquet like I'm Five

What is a columnar format?

Implications of columnar format

Parquet file format

Row Groups

Column Chunks

Supported Types

Building and Working with Parquet

Use Cases

Best Practices

What is Syslog? Components, benefits, and best practices

History and Evolution of Syslog

3 Core Components of Syslog

How Syslog Works

Benefits of Using Syslog

Best Practices for Syslog

Challenges and Limitations of Syslog

Future of Syslog

Conclusion

How to set up a CDC pipeline to capture and analyze real-time database changes with Parseable

Metadata to actionable insights in Grafana: How to view Parseable metrics

Getting Started with Parseable

Instal Grafana and the Parseable plugin

Setting up the Grafana dashboard

Summary

Ingesting Data to Parseable Using Pandas

A Step-by-Step Guide

Requirements

The CSV File

The Parseable API

Writing the Script

Explanation of the Script

Handling Errors and Retries

Next Steps

Build complete website with Docusaurus

Installing Docusaurus

Seeing website in development server

Customizing Home Page

Customizing Navbar

Customizing the footer

Creating a new page

Conclusion

Logging for your Node.js app

Introduction

Table of contents

What is logging?

Using winston in a Node.js application

Conclusion

Reimagine log storage: Parseable

Context

Search Engines

SaaS Platforms

Introduction

Conclusion

Logging for your frontend apps

The Setup

Create a React app

Setup Pino

Add logging to the app

Conclusion