Forem: neon_arch

How to Wrap Your Errors with Enums when using Error-stack

neon_arch — Sat, 01 Jul 2023 17:32:05 +0000

Introduction

I am an intermediate rust developer and you may know me from my posts on subreddits like r/opensource or r/rust or from my various projects on GitHub.

Recently we decided to do error-handling and provide custom error-messages for the errors related to each engine code present under src/engines folder in one of my project websurfx when using error-stack and we wanted to use enums for it and we found that the error-stack project provides no guide, tutorial or example but thanks to one of our maintainers @xffxff, @xffxff provided a really cool solution to this problem and which helped me learn a lot so I decided to share with you all on what I learned from it in this post and how you can wrap errors with enums when using error-stack so stick till the end of the post.

For this tutorial, I will be writing a simple scraping program to scrape example.com webpage and then we will write code to handle errors with enums on the basis of it.

Let's Dive Straight Into It

Simple Scraping Program

Let's first start by explaining the code I have written to scrape the More information href link in the example.com webpage that we will be using throught the program to write error-handling code for it. Here is the code:

//! The main module that fetches html code and scrapes the more information href link and displays
//! it on stdout.

use reqwest::header::{HeaderMap, CONTENT_TYPE, REFERER, USER_AGENT};
use scraper::{Html, Selector};
use std::{println, time::Duration};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // A map that holds various request headers
    let mut headers = HeaderMap::new();
    headers.insert(
        USER_AGENT,
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
            .parse()?, // --> (1)
    );
    headers.insert(REFERER, "https://google.com/".parse()?); // --> (1)
    headers.insert(CONTENT_TYPE, "application/x-www-form-urlencoded".parse()?); // --> (1)

    // A blocking request call that fetches the html text from the example.com site.
    let html_results = reqwest::blocking::Client::new()
        .get("https://example.com/")
        .timeout(Duration::from_secs(30))
        .headers(headers)
        .send()? // --> (2)
        .text()?; // --> (2)

    // Parse the recieved html text to html for scraping.
    let document = Html::parse_document(&html_results);

    // Initialize a new selector for scraping more information href link.
    let more_info_href_link_selector = Selector::parse("div>p>a")?; // --> (3)

    // Scrape the more information href link.
    let more_info_href_link = document
        .select(&more_info_href_link_selector)
        .next()
        .unwrap()
        .value()
        .attr("href")
        .unwrap();

    // Print the more information link.
    println!("More information link: {}", more_info_href_link);

    Ok(())
}

In the Cargo.toml file you will need to provide something like this under the dependencies section:

[dependencies]
scraper="0.16.0"
error-stack="0.3.1"
reqwest = {version="0.11.18",features=["blocking"]}

Now I know the above code might seem intimidating and daunting but you don't need to focus on the implementation because it doesn't matter what the code is, That's why I have numbered the parts that is important for this tutorial.

You might be asking what are the those question marks in the numbered parts?? and what it does??.

What is the Question Mark Operator??

According to our favourite go to resource the rust lang book:

The question mark operator (?) unwraps valid values or returns erroneous values, propogating them to the calling function. It is a unary operator that can only be applied to the types Result and Option

Let me explain in it brief.

The question mark operator (?) in Rust allows to handle any operation that returns a result type with either a value or an error if something bad happens to be handled more graciously like If the operation is completed successfully then the execution of the rest of the function or program continues otherwise it is propogated to the function which called it and exits execution of the rest of the function The propogated error by the function which can then be handled by matching over it by a match statement or if let syntax in the caller function on the other hand if the operation was to be done within the main function then if the operation was to fail then as before the rest of the code is never execuated and the error is then propogated to the standard output (stdout) and it gets dislayed on stdout.

For example, take this rust code:

fn main() {
    match caller("4", "6") {
        Ok(sum) => println!("Sum is: {}", sum),
        Err(error) => println!("{}", error),
    }
}

fn sum_numbers_from_string(
    number_x_as_string: &str,
    number_y_as_string: &str,
) -> Result<u32, Box<dyn std::error::Error>> {
    let number_x: u32 = number_x_as_string.parse()?; 
    let number_y: u32 = number_y_as_string.parse()?;

    println!("This code is being executed!! and the code below will also be executed!! :)");

    Ok(number_x + number_y)
}

Here you can see the main function calls the function sum_numbers_from_string which returns a Result type now if the code with the question mark operator were to error out in sum_numbers_from_string function then the execution will stop right

there and the code below it including the println!() statement will never be executed and a ParseError will be propogated to the main function which will then be matched over by the match statement and the Err handle will be executed.

Now let's take another example by placing the operations with the question mark operator in the main function. The code for which will look something like this:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let number_x: u32 = "4".parse()?;
    let number_y: u32 = "6".parse()?;

    println!("This code is being executed!! and the code below will also be executed!! :)");

    println!("Sum is: {}", number_x + number_y);

    Ok(())
}

Now if the above code were to be executed and the operation with the question mark operator were to fail then as usual the program will stop executing and the error will be propogated to stdout and printed on the terminal.

Note
I know I have missed a lot of finer details but I have done it on purpose for the sake of understanding and simplicity but if you wish to learn more in depth about it then I would recommend reading this blog post.

"Code analyzed and explained"

Writing Code to Handle Errors with Enums

Before we start, let's go briefly over what each operation with question mark operator in each numbered part returns.

For the first numbered part the operation gives a Result type something like this Result<HeaderValue, InvalidHeaderValue> as you now know if this operation fails the InvalidHeaderValue will be propogated by the main function to the stdout and will be displayed on it. Similarly, The second and Third parts return Result types Result<String, ReqwestError> and Result<Selector, SelectorErrorKind> respectively.

Now as we know what each part returns we can start writing code to wrap this errors with enums when using error-stack crate we will first start by writing the error enum and let's call it ScraperError.

#[derive(Debug)]
enum ScraperError {
    InvalidHeaderMapValue,
    RequestError,
    SelectorError,
}

Then we will need to implement two traits on our error enum the Display and Context traits the code for which looks something like this:

impl fmt::Display for ScraperError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            ScraperError::InvalidHeaderMapValue => {
                write!(f, "Invalid header map value provided")
            }
            ScraperError::RequestError => {
                write!(f, "Error occurred while requesting data from the webpage")
            }
            ScraperError::SelectorError => {
                write!(f, "An error occured while initializing new Selector")
            }
        }
    }
}

impl Context for ScraperError {}

By implementing Display trait we provide each error type that will be encountered an approriate error messages and with the implementation of Context trait we give the error enum the ability to be converted into a Report type otherwise if this is not implemented the program results into a compile time error stating that the following cannot be converted into a Report type.

Now we will need to replace each Question mark operator and change the return type of the main function to Result<(), ScraperError> So the code will look something like this:

fn main() -> Result<(), ScraperError> {
    // A map that holds various request headers
    let mut headers = HeaderMap::new();
    headers.insert(
        USER_AGENT,
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?, // --> (1)
    );
    headers.insert(
        REFERER,
        "https://google.com/"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?,
    ); // --> (1)
    headers.insert(
        CONTENT_TYPE,
        "application/x-www-form-urlencoded"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?,
    ); // --> (1)

    // A blocking request call that fetches the html text from the example.com site.
    let html_results = reqwest::blocking::Client::new()
        .get("https://example.com/")
        .timeout(Duration::from_secs(30))
        .headers(headers)
        .send()
        .into_report()
        .change_context(ScraperError::RequestError)? // --> (2)
        .text()
        .into_report()
        .change_context(ScraperError::RequestError)?; // --> (2)

    // Parse the recieved html text to html for scraping.
    let document = Html::parse_document(&html_results);

    // Initialize a new selector for scraping more information href link.
    let more_info_href_link_selector = Selector::parse("div>p>a$")
        .into_report()
        .change_context(ScraperError::SelectorError)?; // --> (3)

    // Scrape the more information href link.
    let more_info_href_link = document
        .select(&more_info_href_link_selector)
        .next()
        .unwrap()
        .value()
        .attr("href")
        .unwrap();

    // Print the more information link.
    println!("More information link: {}", more_info_href_link);

    Ok(())
}

Putting it altogather. The code looks like this:

//! The main module that fetches html code and scrapes the more information href link and displays
//! it on stdout.

use core::fmt;
use error_stack::{Context, IntoReport, Result, ResultExt};
use reqwest::header::{HeaderMap, CONTENT_TYPE, REFERER, USER_AGENT};
use scraper::{Html, Selector};
use std::{println, time::Duration};

#[derive(Debug)]
enum ScraperError {
    InvalidHeaderMapValue,
    RequestError,
    SelectorError,
}

impl fmt::Display for ScraperError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            ScraperError::InvalidHeaderMapValue => {
                write!(f, "Invalid header map value provided")
            }
            ScraperError::RequestError => {
                write!(f, "Error occurred while requesting data from the webpage")
            }
            ScraperError::SelectorError => {
                write!(f, "An error occured while initializing new Selector")
            }
        }
    }
}

impl Context for ScraperError {}

fn main() -> Result<(), ScraperError> {
    // A map that holds various request headers
    let mut headers = HeaderMap::new();
    headers.insert(
        USER_AGENT,
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?, // --> (1)
    );
    headers.insert(
        REFERER,
        "https://google.com/"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?,
    ); // --> (1)
    headers.insert(
        CONTENT_TYPE,
        "application/x-www-form-urlencoded"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?,
    ); // --> (1)

    // A blocking request call that fetches the html text from the example.com site.
    let html_results = reqwest::blocking::Client::new()
        .get("https://example.com/")
        .timeout(Duration::from_secs(30))
        .headers(headers)
        .send()
        .into_report()
        .change_context(ScraperError::RequestError)? // --> (2)
        .text()
        .into_report()
        .change_context(ScraperError::RequestError)?; // --> (2)

    // Parse the recieved html text to html for scraping.
    let document = Html::parse_document(&html_results);

    // Initialize a new selector for scraping more information href link.
    let more_info_href_link_selector = Selector::parse("div>p>a$")
        .into_report()
        .change_context(ScraperError::SelectorError)?; // --> (3)

    // Scrape the more information href link.
    let more_info_href_link = document
        .select(&more_info_href_link_selector)
        .next()
        .unwrap()
        .value()
        .attr("href")
        .unwrap();

    // Print the more information link.
    println!("More information link: {}", more_info_href_link);

    Ok(())
}

"Don't run the above code in excitement or else you will be shocked that it does not work"

But don't be excited just yet because when you run the above code it will throw a scary compile time error as follows:

error[E0599]: the method `into_report` exists for enum `Result<Selector, SelectorErrorKind<'_>>`, but its trait bounds were not satisfied
    |
   ::: /home/destruct/.cargo/registry/src/github.com-1ecc6299db9ec823/error-stack-0.3.1/src/report.rs:249:1
    |
249 |   pub struct Report<C> {
    |   -------------------- doesn't satisfy `_: From<SelectorErrorKind<'_>>`
    |
   ::: /home/destruct/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:503:1
    |
503 |   pub enum Result<T, E> {
    |   --------------------- doesn't satisfy `_: IntoReport`
   --> src/main.rs:77:10
    |
76  |       let more_info_href_link_selector = Selector::parse("div>p>a$")
    |  ________________________________________-
77  | |         .into_report()
    | |_________-^^^^^^^^^^^
    |
    = note: the following trait bounds were not satisfied:
            `error_stack::Report<SelectorErrorKind<'_>>: From<SelectorErrorKind<'_>>`
            which is required by `Result<Selector, SelectorErrorKind<'_>>: IntoReport`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `error-stack-blog` due to previous error

"Decoding the above error"

If you try to decode the error it is very confusing and it doesn't really explain the real problem. The problem with our code is that the error returned from Selector::parse() operation in part three is not thread safe.

Fixing the Thread Safety Issue

To fix the above thread-safety error we will need to map the error of the selector operation of part three to the error-stack Report type by constructing it. Also we will add a custom error message that we want to get it print when this error is encountered.

"Fixing the code"

The code for mapping the error from the operation in part three will look like this:

let more_info_href_link_selector = Selector::parse("div>p>a$")
    .map_err(|_| Report::new(ScraperError::SelectorError))
    .attach_printable_lazy(|| "invalid CSS selector provided")?; // --> (3)

Putting it altogather the whole code looks like this:

//! The main module that fetches html code and scrapes the more information href link and displays
//! it on stdout.

use core::fmt;
use error_stack::{Context, IntoReport, Report, Result, ResultExt};
use reqwest::header::{HeaderMap, CONTENT_TYPE, REFERER, USER_AGENT};
use scraper::{Html, Selector};
use std::{println, time::Duration};

#[derive(Debug)]
enum ScraperError {
    InvalidHeaderMapValue,
    RequestError,
    SelectorError,
}

impl fmt::Display for ScraperError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            ScraperError::InvalidHeaderMapValue => {
                write!(f, "Invalid header map value provided")
            }
            ScraperError::RequestError => {
                write!(f, "Error occurred while requesting data from the webpage")
            }
            ScraperError::SelectorError => {
                write!(f, "An error occured while initializing new Selector")
            }
        }
    }
}

impl Context for ScraperError {}

fn main() -> Result<(), ScraperError> {
    // A map that holds various request headers
    let mut headers = HeaderMap::new();
    headers.insert(
        USER_AGENT,
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?, // --> (1)
    );
    headers.insert(
        REFERER,
        "https://google.com/"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?,
    ); // --> (1)
    headers.insert(
        CONTENT_TYPE,
        "application/x-www-form-urlencoded"
            .parse()
            .into_report()
            .change_context(ScraperError::InvalidHeaderMapValue)?,
    ); // --> (1)

    // A blocking request call that fetches the html text from the example.com site.
    let html_results = reqwest::blocking::Client::new()
        .get("https://example.com/")
        .timeout(Duration::from_secs(30))
        .headers(headers)
        .send()
        .into_report()
        .change_context(ScraperError::RequestError)? // --> (2)
        .text()
        .into_report()
        .change_context(ScraperError::RequestError)?; // --> (2)

    // Parse the recieved html text to html for scraping.
    let document = Html::parse_document(&html_results);

    // Initialize a new selector for scraping more information href link.
    let more_info_href_link_selector = Selector::parse("div>p>a$")
        .map_err(|_| Report::new(ScraperError::SelectorError))
        .attach_printable_lazy(|| "invalid CSS selector provided")?; // --> (3)

    // Scrape the more information href link.
    let more_info_href_link = document
        .select(&more_info_href_link_selector)
        .next()
        .unwrap()
        .value()
        .attr("href")
        .unwrap();

    // Print the more information link.
    println!("More information link: {}", more_info_href_link);

    Ok(())
}

Note
In the above code I have introduced a bug on purpose which will allow us to test whether the error-stack has been implemented successfully with the ScraperError enum.

Running the above code you will see that the code runs as expected it throws a ScraperError and it gives a beautiful error output with the first message we had provided it while implementing Display and the last message which we had mapped it to the error provided by the operation in third part.

Error: An error occured while initializing new Selector
├╴at src/main.rs:77:22
╰╴invalid CSS selector provided

Conclusion

Finally we end this post as we have covered everything that was need to wrap errors with enums when using error-stack crate.

I would love to hear from you:

What new thing you learn't alongside this post??

Also if you found this post helpful then feel free to share it on different social media platforms like twitter, reddit, lemmy, etc.

Contact me on Reddit where I am know by the username u/RevolutionaryAir1922 or message me on Discord where I am know by the username neon_mmd or tag me on Rust Discord server.

If you want to geek out with us. you can join our project Discord server.

Unorganic, Spam, and Ad Results: A Major Issue for Search Engine Users

neon_arch — Sat, 20 May 2023 16:05:46 +0000

Introduction

In today's digital age, search engines have become an integral part of our lives. Whether we're looking for information, products, or services, search engines provide us with a wealth of results. However, alongside relevant and helpful content, there is an increasing presence of unorganic, spam, and ad results. This issue not only hampers the user experience but also affects the credibility and trustworthiness of search engines. In this article, we will delve into the problems caused by unorganic, spam, and ad results and discuss why they are a major concern for search engine users.

1. Unorganic Results: The Struggle for Authenticity

Unorganic search results refer to the content that doesn't naturally appear in search engine rankings based on relevance and quality. This includes websites that manipulate search engine algorithms or use deceptive techniques to gain visibility. Unorganic results make it challenging for users to find accurate and trustworthy information. These results can lead to frustration, wasted time, and even misinformation. Users rely on search engines to provide them with reliable and authentic content, but unorganic results undermine that trust.

2. Spam Results: The Invasion of Unwanted Content

Spam results are another nuisance for search engine users. They often take the form of irrelevant and low-quality websites that aim to deceive or exploit users. Such results can include phishing websites, malicious downloads, or websites filled with excessive ads. Spam results not only waste users' time but can also pose security risks. Search engines constantly battle spam, but some of it inevitably seeps through, causing frustration and raising concerns about privacy and online safety.

a. User-Generated Spam: Misuse of Platform by Users

User-generated spam refers to content generated by individuals who misuse search engine platforms. It can include fake reviews, misleading information, or irrelevant content created to manipulate search rankings. User-generated spam adds to the challenge of finding reliable and accurate information. While search engines implement measures to detect and remove such content, the sheer volume of user-generated spam makes it a persistent issue.

b. AI-Generated Spam: Manipulating Algorithms for Visibility

With the advancement of artificial intelligence, there has been an emergence of AI-generated spam. This type of spam utilizes machine learning algorithms to create content that appears legitimate but lacks substance and relevance. AI-generated spam aims to exploit search engine algorithms, often leading to misleading or unhelpful results. Identifying and filtering AI-generated spam poses a unique challenge for search engines as these techniques constantly evolve.

3. Ad Results: Balancing Revenue and User Experience

While search engines need to generate revenue to sustain their operations, the presence of excessive and intrusive ad results can disrupt the user experience. Ad results are typically displayed prominently at the top of search engine results, sometimes even blending in with organic results, making it difficult for users to distinguish between the two. The overabundance of ads can lead to user dissatisfaction and a sense of information overload. When ad results dominate the search engine page, it can feel like the platform prioritizes monetization over delivering quality and relevant content.

Introducing Websurfx: Combating Unorganic, Spam, and Ad Results

Amidst the challenges faced by search engine users, projects like Websurfx strive to provide a solution. Websurfx is a self-hostable meta search engine that aims to combat unorganic, spam, user-generated spam, and AI-generated spam. The project has set clear goals to address the concerns of search engine users:

Organic and Relevant Results: Websurfx focuses on providing organic search results that are relevant to what users search for. The project does not employ algorithms to promote one content over another, ensuring a fair and unbiased search experience.
Ad-Free and Spam-Free: Websurfx aims to deliver ad-free and spam-free search results. By filtering out unwanted and deceptive content, users can navigate the search engine without distractions and security concerns.
Proper NSFW Blocking: Websurfx takes user safety seriously by implementing proper NSFW (Not Safe for Work) blocking. This feature ensures that inappropriate or explicit content is appropriately filtered, providing a safer browsing experience.
Advanced Image Search: Websurfx offers advanced image search capabilities, allowing users to find relevant images quickly and efficiently.
Dorking Support: Similar to popular search engines, Websurfx supports dorking, enabling users to conduct advanced search queries and fine-tune their search results.
Privacy, Security, and Speed: Websurfx prioritizes user privacy, security, and speed. By minimizing data collection and employing secure protocols, users can search the web with peace of mind.
Quick Results and Micro-Apps: Websurfx aims to provide quick and convenient results. It offers features like a calculator, currency converter, and other micro-apps directly within the search results, saving users time and enhancing their search experience.
AI Integration for Answering Search Queries: Websurfx integrates artificial intelligence capabilities to provide accurate and relevant answers to users' search queries. This feature enhances the search experience by delivering concise information directly within the search results.
High Level of Customizability: Websurfx provides users with a high level of customizability. It allows for easily addable and editable themes, empowering users to personalize their search engine experience. By default, Websurfx offers nine famous colorschemes, providing users with a simple theme selection to suit their preferences.

Although Websurfx is still in its early stages of development, the project holds promise in addressing the issues faced by search engine users. As a small team of developers, Websurfx seeks support and contributions from the community to help grow and make the project a success. By embracing Websurfx, users can say no to spam, ads, and unorganic results and say yes to a cleaner, more reliable search experience.

Conclusion

Unorganic, spam, user-generated spam, and AI-generated spam pose significant challenges for search engine users. They hinder the user experience, compromise trust, and impede the search for reliable information. However, projects like Websurfx strive to combat these issues by providing organic and relevant results, ensuring an ad-free and spam-free environment, implementing proper NSFW blocking, offering advanced search capabilities, prioritizing privacy, security, and speed, providing quick results and micro-apps, integrating AI to answer search queries, and providing a high level of customizability. By supporting and contributing to projects like Websurfx, users can actively participate in creating a better search experience and help shape the future of search engines. Together, we can say no to spam, ads, and unorganic results, including user-generated and AI-generated spam, and say yes to Websurfx.