Forem: Rico van Zelst

How to block ChatGPT from scraping your website.

Rico van Zelst — Tue, 12 Sep 2023 16:36:42 +0000

Introduction

In today's digital day and age, web scraping and parsing have become common techniques for various applications, including data collection, content analysis, and most presently teaching AI models. While these practices can be legitimate and beneficial, there are instances where you don't want a language model trained on your data, leading to concerns about privacy, security, and unauthorized data extraction. In this article, we will discuss how to block ChatGPT (OpenAI), a popular AI language model, from scraping and using your website to train their AI models.

Understanding ChatGPT

ChatGPT is an AI language model developed by OpenAI, known for its ability to generate human-like text based on input prompts. It is widely used for various applications, including chatbots, content generation, and data analysis. ChatGPT can access and parse information from websites through plugins and OpenAIs web crawler, making it essential for website owners who don't want their data trained on by AI models to block its access.

Blocking ChatGPT from your website

Method 1. Robots.txt file

The robots.txt file is a standard method for communicating with web crawlers and bots, including ChatGPT. You can specify which parts of your website are off-limits for crawling by adding rules to this file. To block ChatGPT from your entire website, add the following lines to your robots.txt file:

User-agent: ChatGPT-User
User-agent: GPTBot
Disallow: /

Or if you would like to block ChatGPT only from certain places on your website you can do the following:

User-agent: ChatGPT-User
User-agent: GPTBot
Allow: /can-be-scraped/
Disallow: /will-be-blocked

However, one major downside of the robots.txt method is that it can't actually enforce the instructions. While OpenAI themselves state they will not crawl your website if you disallow their User-agent and I believe this to be true, you can never be sure.

Method 2. Web Application Firewall (WAF)

A web application firewall is a specific form of application firewall that filters, monitors, and blocks traffic to your website. In the case of ChatGPT and its GPTBot web-crawler you have 2 options to block access to your website, you're able to use both if you want to.

The first method is similar to the robots.txt method; blocking its user-agent. OpenAI web crawler will always have the user-agent of "GPTBot" and all ChatGPT plugins will have the user-agent "ChatGPT-User". By blocking the user-agents "GPTBot" & "ChatGPT-User" you will be blocking its access to your website. If you use Cloudflare, here is a guide to user-agent blocking: https://developers.cloudflare.com/waf/tools/user-agent-blocking/
The second method: blocking IP Ranges. OpenAI has published the IP Ranges they use for GPTBot & ChatGPT plugins. You can find them here: https://openai.com/gptbot-ranges.txt. Those are just applicable to OpenAIs web crawler, if you would also like to block the ChatGPT plugins you should also block 23.98.142.176/28. By blocking these IP Ranges you can be confident ChatGPT will be unable to scrape your website, even if they decide to ignore robots.txt or change their user-agent in the future.

Written by: Rico van Zelst

Thank you for reading! If you enjoyed this post and want to explore more of my work, visit my personal website at rico.sh. There, you'll find in-depth articles, tutorials, and resources on web development. Don't miss out – let's continue this journey together!"

The benefits of pair programming for new developers 🚀

Rico van Zelst — Mon, 09 Jan 2023 22:10:01 +0000

As a new developer, it can be intimidating to dive into a project on your own. This is where pair programming can be extremely beneficial. Pair programming is the practice of two developers working on the same task, at the same computer, at the same time. One developer writes the code while the other reviews and offers suggestions. This may sound strange at first, but there are actually several benefits to this approach, particularly for new developers.

Improved code quality

Working with another developer allows you to catch mistakes and discuss best practices, leading to higher quality code. Your partner can point out any errors or areas for improvement that you may have missed on your own. This can help you avoid common pitfalls and ensure that your code is up to standards. In addition, having another set of eyes on your code can help catch any bugs or issues that you may have missed. This can save time and frustration in the long run, as you won't have to spend as much time debugging your code.

Shared knowledge

Pair programming allows you to learn from your partner, expanding your own understanding and skill set. Your partner can share their knowledge and experience with you, helping you grow as a developer. This can be peculiarly useful if you are working with someone who has more experience than you. You can learn from their mistakes and pick up valuable tips and tricks. This can be extremely helpful if you are just starting out in your career and are still learning the ropes. By working with a more experienced developer, you can get a jump start on your own learning and development.

Faster development

Two brains can work through problems faster than one, allowing you to complete tasks more efficiently. By bouncing ideas between one another, you can make far more progress than working alone. This can be helpful when you are under a tight deadline or working on a complex project. This should be able to help you progress faster.

Greater accountability

When working with a partner, you are more accountable for your work and are less likely to procrastinate or take shortcuts. This can help you stay on track and ensure that you are putting in the necessary effort to complete your project. Your partner can also help keep you focused and motivated, which can be especially helpful if you are working on a difficult task. By working with a partner, you are more likely to stay motivated and on task, which can help you get more done in less time.

Increased confidence

Pair programming can boost your confidence as a developer, as you have the support and guidance of another experienced developer. Your partner can provide encouragement and help you overcome any challenges that may arise. This can be specially valuable if you are just starting out in your career and still learning the ropes. By working with a more experienced developer, you can get the support and guidance you need to build your confidence and skills. This can help you feel more capable and competent as a developer, which can be valuable if you are just starting out.

Enhanced communication skills

Pair programming can also help improve your communication skills. Working with another developer requires you to clearly explain your thought process and reasoning behind your code. This can help you improve your ability to communicate technical concepts to others. In addition, by working with a partner, you can learn to listen actively and ask clarifying questions. These skills can be valuable not only in your work as a developer, but also in your overall career.

Overall, pair programming is a valuable tool for new developers. It can help improve code quality, increase shared knowledge, speed up development, promote accountability, and boost confidence. If you have the opportunity to try pair programming, give it a chance – you may be surprised by the benefits it can provide. While it may take some time to get used to working with another person, the long-term benefits are well worth the effort. So, next time you are working on a project, consider giving pair programming a try. You might just find that it helps you become a better developer in the process.

Thanks for reading! If you enjoyed this post, consider following me on GitHub to stay up to date with my latest projects and contributions.

Switching from Windows to Fedora: A Developer's Experience

Rico van Zelst — Sun, 18 Dec 2022 17:17:27 +0000

As a passionate developer, I am constantly looking for ways to improve my workflow and increase my productivity. That's why, after almost a decade of Windows as my primary operating system, I decided to make the switch to Fedora Workstation 37 on my laptop. Here are a few reasons why I made the switch and why I think it's a great choice for other developers.

One of the biggest benefits of using Fedora is the level of customization it offers. With a wide range of desktop environments and tools available, I was able to easily customize the look and feel of my system to suit my preferences. This has been especially helpful as I've been learning the operating system, as I've been able to set things up in a way that works best for me.

One of the main reasons that prompted me to switch to Fedora was the Modern Standby (S0) issues I was experiencing on Windows. Modern Standby is a power-saving mode that is supposed to allow the computer to enter a low-power state while still maintaining connectivity and the ability to perform tasks in the background. However, I found that this feature often caused problems on my machine, such as battery drain. Since switching to Fedora, my battery life in general has significantly improved, and I'm not worried about my laptop being 1% after I closed it at 100% the day before.

In addition to the customization and Modern Standby issues, I've found that using Fedora has helped to increase my productivity. The operating system is stable, fast, and efficient, which allows me to focus on my work without being slowed down by system issues or other distractions. The customizable nature of Fedora has also helped me set up my desktop environment in a way that works best for me, further contributing to my increased productivity.

Another advantage of using Fedora is the package management system, dnf (Dandified YUM). This system makes it easy to install, update, and remove software packages, saving me time and effort compared to manually downloading and installing applications on Windows.

In terms of performance, I've found that Fedora performs just as well, if not better, than Windows on my machine. It's lightweight and runs smoothly, even when I have multiple applications open at the same time. It also uses substantially less RAM than Windows, which is always a bonus.

Finally, I switched to Fedora because I agree with the philosophy behind the operating system. As an open source project, Fedora is built on the principles of collaboration, transparency, and community-driven development. This aligns with my own values and the way I like to work.

In conclusion, there are many reasons why I switched to Fedora from Windows. The level of customization available, the Modern Standby issues I was experiencing on Windows, the increased productivity, the efficient package management system, the excellent performance, and the philosophy behind the operating system all contributed to my decision to make the switch. If you're a developer looking for a reliable and customizable operating system, I highly recommend giving Fedora Workstation 37 a try.

Thanks for reading! If you enjoyed this post, consider following me on GitHub to stay up to date with my latest projects and contributions.

How to get the Fluent Emojis on Windows 10

Rico van Zelst — Mon, 12 Dec 2022 19:10:28 +0000

Microsoft recently introduced their new Fluent Emoji set in the latest update to Microsoft Teams. If you're a fan of these emojis and want to use them across your entire operating system, you're in luck. While the Fluent Emoji set is the default on Windows 11, it is not the case for Windows 10. However, it is easy to change your OS-wide emoji set.

Please note that while this method is unlikely to cause any issues, it is still recommended to back up your Registry and create a System Restore Point before proceeding.

To get started, you will need to download the Segoe UI Emoji font family (Fluent Edition) from here: Click ⏬. And have a basic understanding of how to use Windows.

Once you have downloaded the font, open Command Prompt as an administrator and run the following command:

reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts" /v "Segoe UI Emoji (TrueType)" /f

This step may not always be necessary, but it is recommended for ensuring the success of the process.

Next, open the downloaded font file and click "Install". Once the font is installed, restart your system to complete the process and enjoy the new Fluent Emojis across your entire operating system.

If you want to revert to the Windows 10 emojis at any point, simply download the Windows 10 Segoe UI Emoji font from here ⏬, and follow the steps above again, using the new font file.

Info: I extracted the Segoe UI Emoji (Fluent Edition) .ttf from a Windows 11 VM.

Thanks for reading! If you enjoyed this post, consider following me on GitHub to stay up to date with my latest projects and contributions.