Forem: Jen Wike Huger

Raspberry Pi project: Install project management software

Jen Wike Huger — Tue, 26 Nov 2024 16:01:10 +0000

Did you know OpenProject can run on Raspberry Pi?

OpenProject is a versatile project management platform supporting methodologies like Agile, Scrum, and Kanban while offering classic tools such as Gantt charts and work package management.

In this detailed guide, learn how to install OpenProject on a Raspberry Pi. OpenProject user and author of the tutorial, Giuseppe Cassibba, uses a Raspberry Pi 5 Model B, but this tutorial is adaptable to any 64-bit Raspberry Pi setup.

Here's a short summary.

Requirements and preparation

This guide outlines the necessary hardware, including a Raspberry Pi with a compatible power supply and a high-speed microSD card (minimum 16 GB, Class 10). The installation process relies on Docker for simplicity. It begins by installing the Raspberry Pi OS Lite (64-bit) for a lightweight operating system, updating the system, and setting up Docker.

Installation steps

After preparing the system:

Check OpenProject versions: Identify the latest compatible OpenProject Docker image for the Raspberry Pi's Linux/ARM64 platform.
Run Docker command: A single Docker command initializes OpenProject. The guide explains each part of the command, including setting up environment variables, mapping ports, and creating persistent data volumes to avoid data loss on container restarts.
Monitor initialization: Use docker logs openproject to track the installation progress. The setup typically concludes with the server listening on port 8080.

Access and administration

Once installed, OpenProject is accessible via the Raspberry Pi's IP address and port 8080. Users log in with default credentials (admin/admin) and are prompted to update the admin password. OpenProject's interface includes preloaded example projects to explore its features.

The guide also covers administration tools, including user management, system settings, and email notifications. These allow for extensive customization and configuration of the platform.

For more information on OpenProject, see our docs.

Conclusion

This tutorial highlights the simplicity of hosting OpenProject locally on a Raspberry Pi, ensuring data privacy and control. With Docker, the installation is streamlined and efficient. For further Raspberry Pi projects, explore other tutorials by Giuseppe Cassibba.

Notes from the All Things Open conference USA

Jen Wike Huger — Thu, 31 Oct 2024 14:40:01 +0000

The All Things Open Conference (ATO) was this week!

Attending the open source developer All Things Open Conference always feels like reconnecting with camp friends. This week has been a wonderful reminder of:

⛺ the creativity of open source developers

🔥 the resilience of open source communities

🥾 the humanness of the technology industry

This is a photo from our Friends of Opensource.com lunch for past contributors, including Jason Hibbets, managing editor over at ❤️ We Love Open Source.

I was also reminded that many projects and teams are strapped. 1-2 people often do the job of 4-5 people. OpenProject is an affordable (and open source!) tool that is like adding another person, without the time/money/effort needed to hire someone. It can help get you from A to B right now, bridging the gap while you're getting the time/money/effort together to hire that wonderful human you need.

The All Things Open schedule was filled with keynotes, breakout sessions, and networking opportunities that celebrate and explore advancements in open source technology. It attracted a diverse audience including developers, data scientists, security experts, and community advocates who are passionate about open source innovation, practical implementations, and the collaborative spirit of open communities.

🌞 Day 1 highlights

Morning keynotes included AI-driven code security, inclusivity in tech, and insights from Google’s 20-year support of open source.
Breakout sessions focused on the following themes: DevOps, Linux, Big Data, Cloud, Blockchain, and AI, covering topics like “DevOps and Generative AI,” “Linux growth challenges,” and “AI integration with open source CI/CD.”
Afternoon keynotes focused on solving open source maintenance challenges and exploring inclusivity to boost Linux’s developer base.

🌤️ Day 2 highlights

Morning keynotes focused on security in open source, developer communities, and leveraging LinkedIn for tech networking.
Breakout sessions explored next-generation open source projects, such as secure infrastructure for IoT, scaling cloud infrastructure, GraphQL for APIs, open source AI advancements, and secure coding in the cloud.
Closing remarks by the conference organizer Todd Lewis focused on lasting connections and celebrating the community's growth.

😎 This conference is a must-attend for those who champion open source's collaborative potential and its role in shaping the future of tech!

Project management for software projects – 9 features to consider

Jen Wike Huger — Mon, 21 Oct 2024 18:00:31 +0000

The roles that make up a software engineering project may range from a team of programmers and the team leader to a team with more variety, like developers, UX designer, QA engineer, project manager, and DevOps engineer.

Regardless of whether their roles are similar or different, all of these people must come together to understand the project requirements, constraints, resources, and timeline. They must work together to achieve a result that satisfies the business and customer needs.

Project management features like shared task lists and timelines enable efficient communication across and within these teams.
In this article, learn about the many other features your engineering project management software should offer to help you handle complex workflows, resource management, and technical project requirements.

1. Complex workflows
Engineering projects often involve detailed planning and scheduling. Gantt charts and roadmaps allow teams to break down projects into smaller tasks, set dependencies, and schedule deadlines.
This helps engineers plan long-term projects and keep track of progress. Milestones can be set to track essential stages in the project lifecycle, ensuring the project stays on schedule.

2. Agile (or waterfall or hybrid) methodologies
Engineering teams often use Agile, Waterfall, or hybrid project management approaches depending on the project.
A tool that offers flexibility between the methodologies is ideal for engineering projects with different phases, such as research, design, testing, and implementation.

3. Task and issue management
Engineers need a tool to break down complex engineering projects into smaller, manageable tasks. Then, they can assign specific tasks to team members, set deadlines, and track the progress of each task.

An issue-tracking system is especially useful for identifying and managing problems during the project lifecycle. Engineers can document bugs, defects, or problems in the system, assign them to team members, and track their resolution.

4. Document management and version control
Engineering projects require the management of technical documents, specifications, designs, and blueprints. It's imperative to use a tool or system with document management features that allows teams to store, organize, and share project-related files in a central location.

This feature helps engineering teams maintain version control, ensuring all team members work from the latest design documents, plans, or specifications.

5. Time tracking and cost control
Engineering projects often involve careful management of resources, including time and personnel. You need to be able to log time spent on specific tasks, which is critical for project estimation, cost tracking, and billing.

6. Support for large-scale and long-term projects
Engineering projects often span long periods and involve multiple phases, from research to design, implementation, and maintenance.
Choose software that allows for detailed tracking and reporting over extended periods, ensuring that long-term engineering projects stay on track.

7. Integration with other technical tools
Engineers need to be able to choose their favorite tools for optimal design and development. To keep project management centralized, select a system that integrates with some of the more popular developer tools, like GitHub and GitLab.

8. Data security and privacy
Engineering projects often deal with sensitive data, including proprietary designs, specifications, and research. Self-hosting options allow engineering teams to keep data in-house and secure.
This feature ensures compliance with industry-specific security standards for industries like aerospace, automotive, or IT, where data security is critical.

9. Compliance and reporting
Many engineering fields require adherence to industry regulations and standards. Look for detailed reporting features that allow teams to track progress, ensure compliance with technical and regulatory standards, and generate documentation required for audits or certifications.

Wrapping up

Many project management tools on the market support some or all of the features listed in this article. To choose the right one for you, start by considering the importance of these features to your project and team. If you aren't sure, ask for feedback from others involved in the projects.

For engineering teams, OpenProject provides an all-encompassing platform tailored to the specific needs of projects involved in development and software. With features focused on adaptability, security, and collaboration, engineering teams can efficiently manage short- and long-term projects.

Fullstack Developer, Solutions Engineer Wanted at Remote Tech Company

Jen Wike Huger — Mon, 30 Sep 2024 14:05:32 +0000

OpenProject is special. The culture is 10/10.

I've heard it several times this year after speaking with employees who have been with the company for two years... or ten years. And I've been in the industry since 2012—working for stellar culture companies like Red Hat and difficult ones as well. The difficult companies teach how important it is for your daily mental health to work for a great company that leads with kindness and positive energy.

Like OpenProject, open source project management software for on-prem or cloud. This awesome team has roles open for a Solutions Engineer and a Ruby on Rails Backend Engineer.

Hear what OpenProject employees had to say about them this year:

OpenProject is such a great place. It just clicks. Before OpenProject, I worked for Audi. My new role at OpenProject has been an excellent transition for me to a smaller, more connected team again. - Aaron, a Fullstack Dev

Remote work has always been accepted at OpenProject. Today, I’m working to improve our compatibility with service providers beyond AWS. It's an exciting challenge. - Markus, an SRO Engineer

I have worked on open source projects with people from around the world, so when this role opened up, I was excited. As the only Kenyan on the team, I feel respected and valued. And I respect and value my colleagues. I love to learn about other cultures. - Kabiru, a Software Engineer

We have a small office space in Berlin, and some employees love coming in. But we’re remote first and allow our employees to choose what they prefer. We are as productive as ever and organize offsite meetings to get to know each other better in person. - Birthe, the Chief Marketing Officier

Right fit for you?

If you identify with these characteristics, we'd love to hear from you.

passionate about open source
find joy in working together to find solutions
can accommodate working within European time zones
want a supportive, remote job

Apply for an open role or submit your resume.

20 reasons (and a tutorial!) to deploy OpenProject with Kubernetes via Helm

Jen Wike Huger — Tue, 03 Sep 2024 16:21:57 +0000

Managing your project management software (like, OpenProject) in a cloud-native environment allows you to customize your instance with the added security benefit.

Deploying OpenProject on Kubernetes using Helm offers several advantages, particularly for organizations and teams looking for a scalable, manageable, and efficient way to run their project management software.

20 reasons to deploy OpenProject with Kubernetes via Helm:

Comprehensive Feature Set: OpenProject provides a wide range of features that cater to various aspects of project management. This includes task management, time tracking, project planning, and Gantt charts. It also supports agile methodologies with features like Kanban boards and Scrum management. The tool's flexibility allows it to be used for different types of projects, from simple task tracking to complex project portfolios.

Open Source and Customizable: As an open source platform, OpenProject allows organizations to customize the software to fit their specific needs. This flexibility is a significant advantage for businesses that require tailored solutions. Additionally, because it is open-source, there are no licensing fees, making it a cost-effective option for organizations looking to manage projects without incurring high software costs.

Scalability: Kubernetes can scale OpenProject horizontally (adding more instances) or vertically (increasing resources per instance) based on demand. This ensures that the application can handle varying loads efficiently.

High Availability: Kubernetes can manage multiple instances of OpenProject, ensuring the application remains available even if some instances fail. Helm charts can be configured to deploy OpenProject in a highly available setup, distributing workloads across different nodes.

Simplified Deployment with Helm: Helm is a package manager for Kubernetes that simplifies the deployment process. It allows you to define, install, and upgrade complex applications like OpenProject using charts, which are collections of Kubernetes resources.

Version Control and Rollbacks: Helm tracks the version history of your deployments, making it easy to roll back to a previous version if something goes wrong during an update. This version control is handy in maintaining stable environments.

Consistent Deployments: Helm charts encapsulate all the necessary Kubernetes resources, ensuring that deployments are consistent across different environments (development, staging, production). This Infrastructure as Code (IaC) approach reduces the risk of configuration drift.

Repeatable Deployments: With Helm, you can easily replicate the deployment process across different clusters or environments, ensuring consistency and reducing manual errors.

Parameterization: Helm allows easy customization through values files, where you can override default settings to match your specific requirements. This includes configuring storage, databases, networking, and other critical components of OpenProject.

Centralized Management: By using Helm, you can manage all your Kubernetes resources for OpenProject in a single place, simplifying updates, configuration changes, and monitoring.

Optimized Resource Usage: Kubernetes can optimize resource usage by efficiently scheduling pods across the cluster, ensuring that OpenProject uses only the necessary resources. This helps reduce operational costs, especially in cloud environments where resource utilization directly impacts billing.

Cost Control: With Kubernetes, you can set resource limits and requests for OpenProject, ensuring that it doesn't consume more resources than allocated. This is particularly important for managing costs in a shared environment.

Seamless Integration: Deploying OpenProject on Kubernetes allows easy integration with other cloud-native tools and services, such as monitoring with Prometheus, logging with Elasticsearch, and security management with Vault. Helm charts often include pre-configured options for these integrations.

Multi-Cloud and Hybrid Deployments: Kubernetes supports multi-cloud and hybrid cloud environments, giving flexibility in where and how to deploy OpenProject. Helm facilitates deploying OpenProject across different cloud providers with consistent configurations.

Enhanced Security: Kubernetes provides robust security features like role-based access control (RBAC), network policies, and secrets management. Helm charts can be configured to leverage these features, ensuring OpenProject is securely deployed.

Compliance Management: Kubernetes and Helm allow for detailed control over the deployment environment for organizations with strict compliance requirements, making it easier to meet regulatory standards.

Open Source Ecosystem: OpenProject and Kubernetes are both open source projects with strong community support. Helm charts for OpenProject are often maintained by the community, providing access to updates, patches, and best practices.

Continuous Improvement: By deploying OpenProject on Kubernetes with Helm, you benefit from the continuous improvements and innovations in both the Kubernetes ecosystem and the OpenProject software.

Adapting to Growth: As your organization grows, so will your project management needs. Deploying OpenProject on Kubernetes using Helm ensures that your deployment can scale and adapt over time, avoiding the limitations of traditional, static environments.

Flexibility: Kubernetes and Helm provide a flexible deployment framework that can accommodate changes in infrastructure, technology, or business requirements, helping you stay agile in a rapidly evolving environment.

Tutorial: How to deploy OpenProject with Kubernetes and Helm

We will store attachments via an S3-compatible Object Storage system and automatically generate a TLS certificate via Let's Encrypt. To get started, create a Kubernetes cluster and connect to it with kubectl.

Follow along in this video with our engineering team:

Or, follow along via our GitHub documentation.

All Things Open 2024: What you need to know

Jen Wike Huger — Fri, 30 Aug 2024 16:07:29 +0000

When is the event? 🎃

All Things Open, or ATO, will be held from October 27th to 29th in downtown Raleigh, North Carolina (USA), at the Raleigh Convention Center.

How big is the event? 👀

I hear it will be close to 5k attendees this year.

ATO brings folks from all over the world because of the content and the people. Todd Lewis the event organizer spends a lot of time and effort staying connected to the trends, important topics, and people in open source and tech so that attendees are getting the latest and greatest.

What's the content? Who are the people?

CONTENT

AI is going to be 🔥 everywhere like a wildfire.

Cloudy and code-y stuff will always underpin the trendy stuff.

SPEAKERS

Women are in high numbers on the speaker list, thank goodness.

Lots of folks from places 🎒 like Red Hat, Grafana Labs, MongoDB, Cloudflare and other cloudy places... GitLab and other gity places.

And lots of developers as speakers 🤘

How much is it? $150 for two days 🤑

Seriously? Seriously. REGISTER!.

Plus, there's free content on Sunday - Community and DEI.

Where should I stay? 🌲

The city of Raleigh isn't so big that the location matters terribly, but staying close to the venue is nice if you'd like to walk and socialize before and after the event or set up meetings/hangouts.

I have lived in Raleigh since 2001 and have seen it change dramatically over the past 25 years. This area has several amazing universities and fantastic creatives... like chefs, botanists, yogis, and entrepreneurs of all kinds. This is reflected in fun things to do in nearby towns like Chapel Hill, Durham, Cary, and Wake Forest, to name a few. Downtown Raleigh also has a sampling of great food, nature, yoga, craft beers, etc.

Like I said, you can Uber easily no matter where you stay, and there are some economy options further out, but if you can, I highly recommend that you try to stay in one of the many hotel options downtown. Top rec: Long Leaf Hotel.

Leave me a note in the comment if you want more recommendations or have questions!

How to escalate a problem the wrong way; 4 steps to do it right

Jen Wike Huger — Fri, 30 Aug 2024 15:32:59 +0000

In projects, escalation occurs when one's own or team's understanding is insufficient to resolve an issue.

When something needs to be escalated, unresolved problems, conflicts, or requirements beyond the project manager's or project team's decision-making authority must be passed on to the next level of management... or to another team.

The wrong way to do this is to drop the problem into someone else and walk away.

So, what's the right way?

A timely and appropriate way to handle an escalation includes the following elements:

Escalation Criteria
Escalation Paths
Escalation Process
Follow-up and Feedback

Escalation Criteria

First, a team or department or company should clearly document when and under what conditions an escalation is required.

The realization that escalation is necessary usually results from a comparison of the actual state against the planned state. The plan can be a project plan or any other baseline. It is helpful if the project is managed according to the exception principle and tolerances are established at all levels. This way, it can be determined that an escalation is necessary if a certain type of tolerance is expected to be breached. The types of tolerance may include: time, cost, goals, scope, benefits, risks, and sustainability. Tolerances at all levels of management effectively prevent every minor issue from escalating to the top.

Escalation Paths

An escalation path is a defined chain of individuals or committees to whom issues are escalated, usually from the specialist to the team manager, who escalates to the project manager, and then to the steering committee. From there, it may go up to higher management levels or specialized committees.

Escalation Process

An escalation should be done in writing and as quickly as possible.

Because the recipients of the escalation need to make a decision on how to proceed, the escalation must serve as a decision template. It is not enough to merely deliver bad news; it must be clearly documented:

What is the reason for the deviation from the plan?
What are the consequences in terms of time, cost, goals, scope, benefits, risks, and sustainability?
What can be done? What are the available options?
What are the consequences of these options?
What is the plausible recommendation for the next steps?

A suitable template should be used consistently across all projects.

Follow-up and Feedback

This should be self-evident, but it should be documented in the project contract or project handbook how the mechanism for tracking the progress of the addressed escalation works and how feedback is communicated to the project team.

Do you have any stories how how this was handled the wrong way? The right way?

What is Syslog? Components, benefits, and best practices

Jen Wike Huger — Tue, 30 Jul 2024 13:33:59 +0000

The team at Parseable is writing a ton of content around how to do logging in a lightweight and powerful way. One of their recent installments is about Syslog. Read their blog to learn more.

Syslog is the abbreviation for System Logging Protocol. It is a logging method that allows monitoring and managing logs from different parts of the server or system. The primary use of Syslog data is to trace back errors in case of a failure or event. It can also be used to get information about your system's overall performance.

This Syslog guide will help you understand how it works and its functionality. Let's begin by understanding what it is and how it evolved.

History and Evolution of Syslog

The journey of Syslog began in the early days of the Unix operating system where it served as a simple logging protocol for capturing system messages. Eric Allman created the first instance of Syslog in the late 1980s. Soon it became a must-have tool for Unix administrators. The growing need for a standard logging protocol eventually led to the publication of RFC 3164 the first Syslog standard. It has been further developed and extended into RFC 5424 to establish a more solid and in the meantime flexible basis for the frameworks of message formats that could be used with Syslog.

3 Core Components of Syslog

At its core Syslog consists of several key components and concepts:

Syslog Daemon: The Syslog daemon is responsible for receiving processing and forwarding Syslog messages. Some examples of Daemons include Syslog rsyslog or syslog-ng. It is the central hub for logging activities. Hence it helps capture and appropriately handle all system events along with log data.

Syslog Messages: A typical Syslog message is made up of three parts:

The priority (PRI)
Header
Message (MSG)
The PRI combines facility code and severity indicating the message's source and importance. The header includes a timestamp and the hostname or IP address while the MSG contains the log message or event data.

Facilities and Severities: Syslog categorizes messages by facilities such as kern (kernel messages) user (user-level messages) and mail (mail system). Severities range from emergency (system unusable) to debug (debug messages) allowing administrators to filter and prioritize log data effectively.

How Syslog Works

Syslog operates through a straightforward yet powerful workflow:

Message Flow: Applications generate Syslog messages which are transported over networks using protocols like UDP (User Datagram Protocol) or TCP (Transmission Control Protocol). Upon arrival the Syslog or log server processes and stores these messages according to configured rules.

Syslog Formats: The most common standard Syslog message formats include RFC 3164 and RFC 5424. RFC 3164 is the original Syslog format and is widely supported. The RFC 5424 offers enhanced features including structured data and better timestamp precision.

Benefits of Using Syslog

Several key benefits drive Syslog's widespread adoption.

Centralized Logging: By aggregating logs from multiple sources into a central location Syslog simplifies monitoring and management. This centralized approach enhances visibility and enables more efficient analysis of log events.

Real-time Monitoring: Syslog supports real-time alerting and Syslog monitoring allowing administrators to detect and respond to issues promptly. This capability is crucial for maintaining system uptime and security across network devices.

Compliance and Auditing: Syslog helps organizations meet regulatory requirements by providing a reliable and tamper-evident logging mechanism. It facilitates auditing and records all critical events in the log file.

Troubleshooting: Syslog's detailed and structured logs are invaluable for troubleshooting. By examining log messages administrators can more efficiently identify and resolve issues minimizing downtime and disruption.

Syslog Configuration and Usage
Configuring Syslog involves setting up the Syslog daemon and integrating it with various Syslog tools:

Configuring Syslog Daemon
Basic configuration typically involves specifying log file locations setting log rotation policies and defining message filtering rules. Advanced configurations can include:

Forwarding logs to a central Syslog server
Applying TLS/SSL for secure transmission
Customizing log formats
Common Syslog Tools
Syslog provides the ability to use tools to enhance performance. One of the most famous tools is RSyslog. It is known for its high performance and flexibility and as a result it is widely preferred in modern Linux environments. Syslog-ng offers advanced features like content-based filtering and high-availability clustering. Moreover if you integrate Syslog with log management systems such as Parseable it provides powerful analytics and visualization capabilities.

Best Practices for Syslog

To maximize Syslog's effectiveness administrators should follow these best practices:

Security
Ensure secure transmission of Syslog messages using TLS/SSL. This prevents unauthorized access and protects the integrity of log data.

Performance
Optimize Syslog performance by fine-tuning buffer sizes configuring appropriate log rotation policies and using efficient storage solutions.

Scalability
Plan for scalability by distributing logging workloads across multiple servers and using load-balancing techniques. This ensures that Syslog can handle large volumes of logs in high-demand environments.

Maintenance
Regular maintenance tasks include:

Monitoring disk usage
Checking for log file corruption
Updating Syslog configurations as needed
This proactive approach helps maintain a healthy logging infrastructure.

Challenges and Limitations of Syslog

Despite its benefits Syslog faces several challenges:

Reliability Issues: UDP-based transport while fast can be unreliable leading to potential message loss. Using TCP or implementing reliable delivery mechanisms can mitigate this risk.

Scalability Concerns: Handling large volumes of logs requires careful planning and resource allocation. Implementing distributed logging solutions and efficient indexing can help address scalability challenges.

Complexity: Managing complex Syslog configurations can be daunting. Clear documentation regular audits and leveraging automation tools can simplify configuration management.

Future of Syslog

Syslog continues to evolve adapting to modern IT environments:

Emerging Trends: New developments in logging technologies such as structured logging and enhanced metadata support are enhancing Syslog's capabilities and driving improvements in log analysis and correlation.

Integration with Modern Systems: Syslog is increasingly integrating with cloud-native and containerized environments. This includes support for logging drivers in Docker and Kubernetes enabling seamless logging across diverse infrastructures.

Conclusion

Syslog remains a fundamental tool for system logging offering numerous benefits for IT professionals. Administrators can leverage Syslog to enhance system monitoring compliance and troubleshooting by understanding its history core concepts and best practices. Syslog adapts as the logging landscape evolves ensuring its relevance in modern computing environments.

How to set up a CDC pipeline to capture and analyze real-time database changes with Parseable

Jen Wike Huger — Mon, 22 Jul 2024 16:49:11 +0000

Databases are critical for any application. Data constantly gets updated, inserted, and deleted. In most of cases it is important for the business to keep track of these changes due to security concerns, auditing requirements, and to keep other relevant systems up to date.

Change Data Capture (CDC) has gained popularity, precisely to address this problem. CDC is a technique used to track all changes in a database and capture them in destination systems. Debezium is a popular CDC tool that leverages database logs as the source of truth, and streams the changes to Kafka and compatible systems like Redpanda.

The key thing to note here is that while databases are typically mutable, the changes captured by the CDC system should be immutable. Only then users have a system of record that can be trusted. Parseable is built for immutable logs and streams of events and data. It is a great fit for CDC targets. The way we designed Parseable allows for long term retention and querying of logs, and it also provides a rich set of tools for analysis and monitoring.

In this tutorial, we use Redpanda, PostgreSQL, and Debezium to ingest CDC events into Parseable.

Metadata to actionable insights in Grafana: How to view Parseable metrics

Jen Wike Huger — Thu, 11 Jul 2024 15:55:59 +0000

Parseable deployments in the wild are handling larger and larger volumes of logs, so we needed a way to enable users to monitor their Parseable instances.

Typically this would mean setting up Prometheus to capture Parseable ingest and query node metrics and visualize those metrics on a Grafana dashboard. We added Prometheus metrics support in Parseable to enable this use case.

But we wanted a simpler, self-contained approach that allows users to monitor their Parseable instances without needing to set up Prometheus.

This led us to figuring out a way to store Parseable server's internal metrics in a special log stream called pmeta.

This stream keeps track of important information about all of the ingestors in the cluster. This includes information like the URL of the ingestor, commit id of that ingestor, number of events processed by the ingestor, and staging file location and size.

This is a sample event in the pmeta stream.

{
  "address": "http://ec2-3-136-154-35.us-east-2.compute.amazonaws.com:443/",
  "cache": "Disabled",
  "commit": "d6116e8",
  "event_time": "2024-07-02T09:49:05.125255417",
  "event_type": "cluster-metrics",
  "p_metadata": "",
  "p_tags": "",
  "p_timestamp": "2024-07-02T09:49:05.540",
  "parseable_deleted_events_ingested": 35095373,
  "parseable_deleted_events_ingested_size": 10742847195,
  "parseable_deleted_storage_size_data": 1549123461,
  "parseable_deleted_storage_size_staging": 0,
  "parseable_events_ingested": 3350101,
  "parseable_events_ingested_size": 1054739567,
  "parseable_lifetime_events_ingested": 38445474,
  "parseable_lifetime_events_ingested_size": 11797586762,
  "parseable_lifetime_storage_size_data": 1732950386,
  "parseable_lifetime_storage_size_staging": 0,
  "parseable_staging_files": 2,
  "parseable_storage_size_data": 183826925,
  "parseable_storage_size_staging": 0,
  "process_resident_memory_bytes": 113250304,
  "staging": "/home/ubuntu/parseable/staging"
}

Let's show you how to visualize this data in a Grafana dashboard.

We'll start by setting up Parseable to collect this pmeta data.

Getting Started with Parseable

Parseable is a cloud-native log management solution that efficiently handles large-scale log data. By integrating Parseable with your infrastructure, you can streamline log ingestion, storage, and querying, making it an essential tool for observability and monitoring.

You can choose the right installation process for you.

To quickly install Parseable using Docker, open the terminal and type the command:

docker run -p  8000:8000 \  
-v /tmp/parseable/data:/parseable/data \  
-v /tmp/parseable/staging:/parseable/staging \  
-e  P_FS_DIR=/parseable/data \  
-e  P_STAGING_DIR=/parseable/staging \  
containers.parseable.com/parseable/parseable:latest \  
parseable local-store

You can verify the installation by accessing the Parseable UI by navigating to http://localhost:9000 in your web browser. Log in using the default credentials (admin/admin) and explore the dashboard to ensure everything is set up correctly.

Finally, we need to create a log stream before we can send events. A log stream is like a project that will essentially store all your ingested logs. For this tutorial, we'll have a log stream named pmeta. To create a log stream, log in to your Parseable instance, and you'll find a button on the right-hand top side.

Note that:

pmeta is automatically created and populated in a Parseable cluster (high availability setup)
pmeta is not created in a single node setup
if you're not interested in this data, you can set the retention to 1 day for the pmeta stream to avoid storing this data

Read more about the pmeta stream in the Parseable documentation.

Instal Grafana and the Parseable plugin

Grafana helps you collect, correlate, and visualize data with beautiful dashboards. We'll connect the parseable instance with Grafana via the Parseable Grafana datasource. This plugin allows you to query Parseable data using SQL and visualize it in Grafana.

If you want to self-host Grafana, you can host it on a dedicated cloud instance or locally, depending on your requirements. Follow the official Grafana installation guide for more information.

Once the Grafana instance is setup, let's quickly install the Parseable plugin and connect our Parseable instance to Grafana.

Login to your Grafana instance and navigate to the administration setting on the left-hand side menu.
Click on Plugin and Data Option

Open the Plugins page and search for Parseable

Install the plugin and then click on Add New Datasource

From the datasource page, fill in the following details:

In the URL field, type the URL of your Parseable query instance. For example, https://demo.parseable.com:443.

Under the Auth Section, switch to the Basic Auth setting.

In the Basic Auth Details section, enter your Parseable username and password.

Finally, click on Save & Test to verify the connection.

Setting up the Grafana dashboard

We'll now use the Parseable Data Source to query data from the Parseable Meta Stream (pmeta).

Navigate to the Dashboard section and click on New and select Import Dashboard.

Enter the Dashboard ID as 21472 and click on load.

Ensure the data source to Parseable-DataSource by selecting from the dropdown menu.

Once done, click on import. It should take a few seconds to load, and then it will create the dashboard.

We query data from Parseable using SQL. To learn more about querying data in Parseable, you can refer to our documentation.

Summary

Now, you've learned how to create a Grafana dashboard using Parseable's pmeta stream.

This dashboard provides crucial insights into your Parseable instance's performance, and we encourage you to customize this dashboard further to fit your specific needs.

🏃🏽‍♀️ To see Parseable in action, watch this YouTube video. Get started with Parseable in just a single command.

💬 For technical questions or to share your implementations, join our developer community on Slack.

📝 Read more from the Parseable blog.

Ready to enhance your observability? Start using Parseable and Grafana today to unlock the full potential of your log data.

Ingesting Data to Parseable Using Pandas

Jen Wike Huger — Wed, 26 Jun 2024 18:40:04 +0000

A Step-by-Step Guide

Managing and deriving insights from vast amounts of historical data is not just a challenge but a necessity. Imagine your team grappling with numerous log files, trying to pinpoint issues. But, because logs are stored as files, it is very inefficient to search through them. This scenario is all too familiar for many developers.

Enter Parseable, a powerful solution to analyze your application logs. By integrating with pandas, the renowned Python library for data analysis, Parseable offers a seamless way to ingest and leverage historical data without the need to discard valuable logs.

In this post, we explore how Parseable can revolutionize your data management strategy, enabling you to unlock actionable insights from both current and archived log data effortlessly.

Requirements

Python installed on your system
Pandas library
Requests library
A CSV file to be ingested
Access to the Parseable API

The CSV File

Our code example is based on a Kaggle Dataset. We've used a CSV file named e-shop_clothing_2008.csv. Feel free to use your own dataset to follow along. First, ensure your CSV file is formatted correctly and accessible from the script's directory.

The Parseable API

Next, we'll interact with the Parseable API to send our data. Here we're using the demo Parseable instance. Before sending any data, please ensure you have entered the correct endpoint and credentials:

Endpoint: https://demo.technocube.in/api/v1/ingest
Username: admin
Password: admin

Writing the Script

Here’s a Python script that reads the CSV file in chunks and sends each chunk to the Parseable API (in a stream called testclickstream). Replace the CSV file path, Parseable endpoint, and authentication credentials with your own.

import pandas as pd
import requests
import json

# Define the CSV file path
csv_file_path = 'e-shop_clothing_2008.csv'

# Define the Parseable API endpoint
parseable_endpoint = 'https://demo.technocube.in/api/v1/ingest'

# Basic authentication credentials
username = 'admin'
password = 'admin'

headers = {
    'Content-Type': 'application/json',
    'X-P-Stream': 'testclickstream'
}

# Read and process the CSV file in chunks
chunk_size = 100  # Number of rows per chunk
for chunk in pd.read_csv(csv_file_path, chunksize=chunk_size, delimiter=';'):
    # Convert the chunk DataFrame to a list of dictionaries
    json_data = chunk.to_dict(orient='records')

    # Convert list of dictionaries to JSON string
    json_str = json.dumps(json_data)

    # Send the JSON data to Parseable
    response = requests.post(parseable_endpoint, auth=(username, password), headers=headers, data=json_str)

    # Check the response
    if response.status_code == 200:
        print('Chunk sent successfully!')
    else:
        print(f'Failed to send chunk. Status code: {response.status_code}')
        print(response.text)

Explanation of the Script

We have divided the code's flow into six steps to help you understand its function. This will also help you understand how Pandas libraries and Parseable work together.

Importing Libraries
The script starts by importing Pandas for data manipulation, Requests for HTTP requests, and JSON for handling JSON data.
Defining File Path and Endpoint: Specify the path to the CSV file and the Parseable API endpoint. Replace these with your actual file path and API endpoint.

Authentication and Headers
Set up basic authentication credentials and headers. The X-P-Stream header indicates the stream or collection name.
Reading CSV in Chunks: Use pd.read_csv to read the CSV file in chunks of 100 rows. The chunk size parameter handles large files efficiently without memory issues.

Converting Data to JSON
Convert each chunk to a list of dictionaries using to_dict with orient='records', then to a JSON string.

Sending Data to Parseable
Send the JSON data to the Parseable API using a POST request. Check the response status code to ensure successful ingestion. Print any errors.

Handling Errors and Retries

Network issues or server errors might prevent successful data ingestion in real-world scenarios. To make the script more robust, implement error handling and retries. Also, look for code errors, if any.

Next Steps

Ingesting data into Parseable using Pandas is straightforward and efficient. By reading data in chunks and converting it to JSON, we can seamlessly send it to the Parseable API.

This script serves as a foundation and is customizable to your specific needs, including sophisticated error handling, logging, or parallel processing.

Follow this guide to integrate Pandas and Parseable effectively, ensuring smooth and efficient data ingestion for your projects.

To get started or try Parseable, visit our demo page.

Dear engineer, stop trying to do marketing

Jen Wike Huger — Fri, 21 Jun 2024 14:35:42 +0000

When you're starting out, it's easy to just fly by the seat of your pants. A blog here, a social post there. You write some, your engineers or community members write some.

Good, you're starting. And part of that is just doing.

But then, you get funding. Hire some folks. Start to coordinate efforts.

👽 YOU NEED A CONTENT MANAGER 👽

Leaving this up to yourself or your engineers creates disorganized and, thus, ineffective efforts. It's all wasted if it's not coordinated from ideation to writing and editing to final reviews and publishing.

I did this for Red Hat's Opensource.com site for 10 years and genuinely love bringing the process to teams and companies. Startups especially need this process and strategy as they get things up and running for the first time, but all companies can benefit from a content or marketing reset or refresh.

Times in tech have been especially hard with layoff after layoff, particularly impacting marketing-content-community teams. But one thing I've learned over the years, engineers cannot make it in the game of business without us. You may have figured out how to engineer software to solve an important problem, but if you don't set up your show-and-tell side of the house with some serious know-how and process, it will all fall by the wayside... getting you nowhere fast.

Why does this matter? Competition is fierce.

And what I can promise you is that you don't really understand how the following works well enough to do it yourself:

write compelling content consistently (nailing the tone, audience, links)
pitch your content and company (to publications, podcasts, webinars)
create and maintain partnerships with similar businesses/integrations (co-webinars/gigs/blogs)
manage your community of customers and contributors (real engagement with real humans overtime that makes an impact)

I highly recommend you read this article by the team at Plane.

⚡️ YOU NEED A 30-60-90 DAY CONTENT STRATEGY PLAN ⚡️

Here's how it works:

🎧 Get on a call with an expert about where you are on your journey regarding content, customers, and community. Doesn't have to be a big deal. They ask the questions, you just give the real answers.

✍ They go back to their desk to make a plan.

This is my 4-step process.

1️⃣ Review and research any materials you've shared with me, your website and other content, your audience and industry... all that good stuff.

2️⃣ Create a 30-60-90 day content strategy plan (or general marketing plan).

3️⃣ Add a healthy dose of structure and process: I build content calendars and editorial workflows for you.

4️⃣ Delivery! You and your team are set up to execute a stellar new content plan.