Forem: Vikram Aruchamy

[Boost]

Vikram Aruchamy — Fri, 04 Jul 2025 12:29:30 +0000

Vikram Aruchamy

Jul 4 '25

ChatGPT to Google Doc

#chatgpt #googledocs #convert #equations

Comments

5 min read

ChatGPT to Google Doc

Vikram Aruchamy — Fri, 04 Jul 2025 12:28:00 +0000

Maintaining a clear and organized record of your ChatGPT conversations is essential, especially when they contain important summaries, complex code, or detailed explanations.

While ChatGPT provides a native copy button for individual responses, it copies the content in Markdown format. When the copied Markdown is pasted into Google Docs or Word, the formatting is often lost or inconsistent. Additionally, there’s no built-in option to copy or export an entire conversation—making it tedious to save longer threads or full sessions.

You can convert ChatGPT content into Google Doc using the ChatGPT to Google Docs or PDF Chrome Extension.

This powerful tool simplifies the process of exporting ChatGPT responses into well-formatted Google Docs, Word, or PDF files, making them easy to share, store, or further edit.

Let’s explore how this extension can help streamline your workflow.

Installating the Extension

The ChatGPT to Google Docs or PDF Chrome extension can be installed in just a few clicks. Follow the steps below to get started on Google Chrome or Microsoft Edge.

Installing on Google Chrome

Open the Chrome Web Store listing.
Click the "Add to Chrome" button.
Confirm the installation by clicking "Add extension" in the popup.
You should now see the extension icon in your browser toolbar.

Installing on Microsoft Edge

Even though this is a Chrome extension, Microsoft Edge supports installing extensions directly from the Chrome Web Store.

Open the Chrome Web Store listing in Microsoft Edge.
If prompted, click Allow to enable installation of extensions from other stores.
Then click "Add to Chrome" and confirm the installation.

Once installed, the extension works the same way in both browsers.

Refresh ChatGPT after Installing Extension

After installation:

Open ChatGPT and refresh the page.
You’ll now see "Google Docs", "Word" and "PDF" buttons appear below each response and at the top-right of the conversation.

Exporting Single Response From ChatGPT into Google Doc

When you want to save just one specific response, such as a code snippet, a definition, or a key insight, from a longer ChatGPT conversation, the extension lets you export it instantly into a Google Doc.

To convert ChatGPT's single response into a Google Docs, follow the steps:

Open any conversation on ChatGPT.
Navigate to the individual response you wish to export.
Look for the Google Docs, Word, and PDF buttons conveniently located at the bottom right corner of each response.
Click the Google Docs button. The content from the single response will be exported into a Google Docs and the output document will open automatically.

What gets preserved?

The exported file maintains the original formatting, saving you from tedious manual adjustments:

Headings and paragraph structure
Bullet points and numbered lists
Code blocks and Markdown formatting
Math equations (rendered perfectly in Google Docs, Word and PDF)
Tables and indentation

No extra formatting required – the document is ready to use or share.

Tip: This is perfect for quickly saving helpful code snippets, or concise explanations without the hassle of copying and pasting.

You can improve clarity and structure by grouping the two export options under clearly labeled subheadings and using parallel phrasing. Here's a more polished and professional version without changing the meaning:

Exporting Entire ChatGPT Conversation to Google Doc

Whether you're documenting a research session, project discussion, or keeping a complete record of your AI interactions, exporting the entire conversation can be invaluable. The extension gives you flexibility to include or exclude your own prompts.

Option 1: Exporting entire Conversation with User Prompts

To export the full conversation, including both your prompts and ChatGPT’s responses,follow the below steps:

Open any ChatGPT conversation at chat.openai.com.
Locate the Google Docs, Word, or PDF button at the top right corner of the conversation window.
Click the Google Docs button to export the entire conversation into a Google Doc.

Option 2: Export Only ChatGPT Responses (Excluding User Prompts)

If you want to save only the AI’s output—useful for compiling answers, articles, or summaries without your input, you can disable prompt export by following the below steps:

Click the Settings icon at the top right of the ChatGPT conversation page.
In the Settings popup, enable the checkbox labeled Export only ChatGPT responses under the Conversion Settings section.

Note: This setting is saved automatically and will apply to all future exports until you disable it.

Exporting ChatGPT Canvas Content into a Google Doc

For those utilizing ChatGPT's Canvas mode in GPT-4 or team accounts, the extension also provides a dedicated export feature for your visual and structured content.

How to export Canvas content:

Maximize the canvas so it's fully open on your screen.
Once maximized, look for the Google Docs, Word, or PDF button in the top right corner of the canvas.
Click the Google Docs button to export the Canvas content into a clean Google Doc.

Important Behavior:

If the Canvas is open, clicking the Google Docs button available at the top will export only the Canvas content.
If the Canvas is closed, the same buttons will revert to exporting the entire conversation.

What gets exported?

When exporting from Canvas mode, the extension ensures:

Only the Canvas content is exported.
The structure, formatting, and any math equations (if present) are perfectly preserved.

Tip: This feature is invaluable for saving structured diagrams, mind maps, or other visual outputs created within Canvas mode, ensuring you capture precisely what you've designed.

Problems Solved by the ChatGPT to Google Docs or PDF Chrome Extension

The ChatGPT to Google Docs or PDF Chrome Extension addresses two common pain points:

It allows you to export content from ChatGPT into Google Docs without losing formatting, preserving headings, lists, code blocks, and more.
It allows you to copy and paste math equations from ChatGPT into Google Docs, rendering them accurately without manual adjustments.

This ensures clean, structured documents, ready to share, edit, or publish.

Conclusion

The ChatGPT to Google Docs or PDF Chrome Extension is an essential tool for anyone who regularly works with ChatGPT content.

It streamlines the process of exporting responses—eliminating the need for manual copying, fixing formatting issues, or dealing with broken math equations.

With just a click, your AI-generated content is seamlessly transformed into a clean, structured Google Doc or PDF, ready for sharing, editing, or archiving.

Whether you're saving summaries, code, research notes, or equations, this extension simplifies your workflow and enhances productivity. Try it out and experience a more efficient way to manage your ChatGPT interactions.

Export Google Docs to PDF, But Smarter: Merge, Protect, and Export with Ease

Vikram Aruchamy — Tue, 21 Jan 2025 12:35:36 +0000

Exporting Google Docs to PDF is a common requirement for professionals, students, and businesses. Google Docs provides a built-in method to convert documents to PDF format, which can be accessed through the "File" menu. Here’s how you can do it using the native method:

Open your Google Docs document.
Click on File > Download > PDF Document (.pdf).
The document will be downloaded in PDF format to your local storage.

While this method is straightforward and effective for basic document exports, it comes with several limitations.

Limitations of Native Google Docs PDF Export

Despite its simplicity, the built-in PDF export feature in Google Docs has some significant drawbacks:

Only the first tab is exported - Google introduced the multi-tab feature in Google Docs, allowing users to organize content better. However, the native PDF export function only captures the content from the first tab, ignoring all subsequent tabs. If you have content organized in multiple tabs, they are not included.
No password protection - Google Docs does not offer any options to secure the exported PDF with a password, which can be a critical requirement for sensitive documents.
No option to merge multiple Google Docs into a single PDF - If you have multiple documents that need to be combined into one PDF file, you must download them separately and use external tools to merge them, which can be time-consuming and inefficient.

These limitations can pose challenges for users who require more advanced document handling capabilities. This is where Docs to PDF Pro comes in.

Introducing Docs to PDF Pro

Docs to PDF Pro is a powerful Google Docs add-on that overcomes the limitations of the native export feature and provides a comprehensive solution for all your PDF conversion needs. Here’s what it offers:

Merge Multiple Google Docs into a Single PDF - Select multiple Google Docs and combine them into one cohesive PDF document.
Export All Tabs in a Multi-Tabbed Google Doc - Ensure that all tabs within your Google Docs file are included in the final exported PDF.
Password-Protected PDF Export - Add an extra layer of security by applying password protection to your exported PDFs.
Local File Upload and PDF Conversion - Convert local Word or text files to PDF directly from within the Google Docs interface.

How to Export Google Docs to PDF Using Docs to PDF Pro

Installation Steps

Install the Docs to PDF Pro add-on from the Google Workspace Marketplace.
Open your Google Docs document and navigate to Extensions > Docs to PDF Pro > Convert Docs to PDF option.
Choose the desired options, such as merging multiple documents, exporting all tabs, or adding password protection.
Click on Export to generate your customized PDF.

Features Explained

Once the add-on is launched, you will see the following options:

Conversion Settings (Optional):

Include tab names as pages when exporting multi-tabbed documents - Ensures that each tab's name in your document is included in the exported PDF as separate pages. This helps you identify to which the content belongs to.
Merge all selected documents into a single PDF - Allows combining multiple Google Docs into one comprehensive PDF file.

PDF Password Protection (Optional):

Enable password protection for the output PDF - Secure your confidential Google Docs by exporting them as password-protected PDFs. Enter a password in the provided field to safeguard sensitive documents like contracts or legal agreements during export.

Output Destination:

Download PDF files and save a backup to Google Drive - Automatically saves a copy of the PDF to your Google Drive while downloading it locally.
Save PDF files only to Google Drive - Stores the generated PDF exclusively in your Google Drive.
Download PDF files only - Downloads the PDF directly to your computer without saving it to Google Drive.

After selecting the appropriate settings, you can choose from the following actions:

Convert Current Document to PDF - Exports the currently open Google Docs into PDF format
Select Multiple Documents & Convert to PDF - Allows selection of multiple Google Docs for converting them to PDF based on the selected settings.
Upload Local File(s) & Convert to PDF - Enables conversion of locally stored documents to PDF format.

With Docs to PDF Pro, you can unlock the full potential of your document exports, ensuring efficiency, security, and flexibility.

Conclusion

While the native Google Docs export feature is useful for basic needs, it falls short when it comes to handling complex document requirements.

Docs to PDF Pro offers an all-in-one solution to address these shortcomings, making it easier than ever to merge, protect, and export Google Docs with ease. Whether you're dealing with multi-tabbed documents or need secure PDFs, this add-on is the perfect companion for your workflow.

How to Convert Markdown to Google Docs

Vikram Aruchamy — Tue, 30 Apr 2024 09:01:11 +0000

Markdown's simplicity and readability make it a popular choice for drafting content, but when it comes to sharing or collaborating on documents, Google Docs offers powerful collaboration features and accessibility.

You can convert Markdown into Google Docs using the Docs to Markdown Pro add-on.

In this guide, I'll show how to convert Markdown files into Google Docs using the Docs to Markdown Pro add on with formatting.

Installing Docs to Markdown Pro

You can visit the Docs to Markdown Pro page in Google Workspace Marketplace and click Install.

The add on will be installed in you account.

Converting Markdown to Google Docs using Docs to Markdown Pro

To convert the Markdown into Google Docs, first launch the converter by selecting the Menu Extensions -> Docs to Markdown Pro -> Convert Markdown or HTML to Docs option as shown in the following image:

The Markdown to Google Docs converter dialog will be displayed as follows.

Now to convert the Markdown into Google Docs, follow the below steps:

Paste the Markdown into the input text box
Select the desired target Google Docs - Whether to insert at the end of the current Document or to create a new Google Docs with the Markdown content
Click Convert.

The Markdown will be converted into Google Docs and inserted into the current document or a new document based on the selected input.

All the styles including tables, table borders and the images are supported in the add on.

Docs to Markdown Pro - Other Features

Docs to Markdown Pro is a comprehensive add-on that bridges the gap between Google Docs and Markdown.

It has all the features needed for the users who work with both Google Docs for collaborative writing/editing and Markdown for ease of writing.

Converts Google Docs to Markdown
Converts Markdown to Google Docs
Converts Google Docs to Clean HTML
Converts HTML to Google Docs
Publishes the Google Docs as Markdown to GitHub/GitLab

Some Usecases of the Markdown to Google Docs

This addon helps you in the following use case where you need to convert Markdown into Google Docs.

Copy a Markdown from ChatGPT to Google Docs
Copy a Markdown from Gemini to Google Docs
Copy a Markdown from CoPilot to Google Docs

It also helps you paste the ChatGPT Markdown into Google Docs without background.

Can Google Docs open a Markdown file?

Yes, Google Docs can open the Markdown file. But the Markdown file must be available in the Google Drive already to open it in Google Docs. However, the text will still be in the Markdown format. It will NOT be converted into the rich text format.

If you need to open the Markdown file in Google Docs and the text in the rich text format, you need to insert the Markdown into Google Docs using the add-on Docs to Markdown Pro.

If you have any questions, feel free to comment below.

Wordable Alternative

Vikram Aruchamy — Tue, 30 Apr 2024 06:24:04 +0000

Wordable.io is a tool that helps users publish Google Docs to WordPress.

Docs to WP Pro is the best alternative to Wordable, because of its features, ease of use and affordable pricing.

In this post, I’ll explain the Docs to WP Pro features and why it is better when compared to Wordable.

Docs to WP Pro vs Wordable

Features	Docs to WP Pro	WordPress
Export directly from Google Docs	Yes	No
Number of clicks to publish	1	> 5
Classic editor support	Yes	Yes
Gutenberg editor support	Yes	No
Semantic internal links	Yes	No
Accessibility checks	Yes	No
Reading time info	Yes	No
Featured image support	Yes	No
Image compression	Yes	Yes
Image alt text	From Google Docs image properties	Add a separate tag, before the image
Readability score	Yes	No
SEO checks	Yes	No
Focus Keyword	Yes	No
Yoast support	Yes	No
RankMath support	Yes	No
Default Taxonomies	Yes	No
New tag addition	Yes	No
Custom post types	Yes	No
Separate interface for publishing	No, one-click publishing from Google Docs	Separate interface
Pricing starts from	$ 39 per year, for up to 400 WordPress exports. No limits on the number of posts per month. Also, supports two WordPress sites.	$ 29 per year, for up to 120 posts, capped at 10 posts per month. Only one WordPress site allowed

Features of Docs to WP Pro - A Wordable Alternative

Docs to WP Pro streamlines publishing Google Docs to WordPress and optimizes your content with data-driven features, including:

Safe and Secure - Data and credentials are encrypted using a 32-bit encryption key and stored securely using Google's industry-leading security measures. Your unique 32-bit encryption key is protected within the User Properties service, ensuring your information is completely isolated and accessible only to you.

Asks for least possible oauth permission - Docs to WP Pro asks to access only the current document you work on. It doesn't ask to access all your Google Docs documents or the Google Drive files.

SEO Suggestions - Checks all the necessary SEO suggestions as similar to Yoast or RankMath, including focus keyword placement in the title, content, keyword density, content readability, alt text in images, meta descriptions, and all the necessary SEO best practices, and suggests corrections.

Semantic internal links using AI - With a single click, Docs to WP Pro can find the potential keyword in your Google Docs content and create appropriate internal links. It helps in improving the SEO of your content.

Link suggestions with contextual data - It also suggests internal links with useful data such as relevance score, existing number of internal links, published date and update date. This helps in adding highly relevant posts as Related posts and the data assists in balanced internal link distribution.

Image Compression - Optionally, it allows you to compress the images before publishing them to the WordPress media library and helps keep the media library sleek.

Yoast and RankMath Support - Allows you to set focus keywords and meta descriptions and update them to the Yoast or the RankMath fields in WordPress. This keeps the Yoast or Rank Math data consistent for posts, regardless of whether you publish from Google Docs or the WordPress dashboard.

Custom Post Types - It helps you publish not only WordPress posts or Pages but any type of custom posts you create on your site.

Classic & Gutenberg Editor Support - Compatible with both Classic and Gutenberg WordPress editors. If your site uses the Classic editor, the content is published as plain HTML. If it’s using Gutenberg, the content is published as Gutenberg blocks. The only plugin that supports the creation of Gutenberg blogs.

Content Link Tracking Using Google Sheets - When you publish a post, an entry is published to a consolidated Google Sheets. This way it is easy for you to check which Google Docs is Published to Which WordPress post. You can directly open the Google Docs from this sheet to update the content and push the update to WordPress in a single click.

With all these features at affordable pricing, the Docs to WP Pro add-on is the best alternative for Wordable.io.

How to Convert PDF File into a Google Docs

Vikram Aruchamy — Sat, 27 Apr 2024 05:38:08 +0000

A PDF (Portable Document Format) file is a versatile document format that preserves the layout and formatting of a document, making it ideal for sharing and viewing across different platforms and devices.

However, it cannot be edited directly. Hence, to make any updates or modifications, you must convert it to Google Docs.

Converting PDF files to Google Docs allows for easy editing, collaboration, and sharing in real time, enhancing productivity and workflow efficiency.

In this tutorial, I'll show you how to convert a PDF into a Google Docs without loosing the formatting using the PDF to Docs Editor Add-on.

The add-on allows you to convert a local PDF file or a PDF file available in Google Drive into Google Docs.

Installing the PDF to Docs Add-on

To install the add-on, navigate to the PDF to Docs app page and click Install. It asks for the necessary permissions. Click Allow to complete the installation. The reason why each permission is necessary is explained in the next section.

Once the add-on is installed, launch the add-on using the menu Extensions -> PDF to Docs -> Convert PDF to Docs.

Add on Permissions

See, edit, create, and delete all your Google Docs documents - This permission is necessary to be able to read the PDF files available in Google Drive when you choose to convert the PDF files available in Google Drive into Google Docs.
See, edit, create, and delete only the specific Google Drive files you use with this app - This permission is necessary to insert the PDF file into the currently open document, in case you choose the insert option.

All the other permissions are typical permissions to install and run an add-on inside the Google Docs editor.

PDF to Docs Add on Features

Create a new document from the PDF content - Select this option to create a new Google Docs based on the PDF file. This option is the default method and is suitable for bulk conversion of PDF into Google Docs.
Insert PDF content at the end of the current document - Select this option to insert the contents of the PDF file into the currently open Google Docs. However, this option works only when you select a single file for conversion.
Open output document in new tab(s) - Select this option if you want to open the newly created Google Docs automatically in a new tab.

Converting PDF files from Local Drive into a Google Docs

To convert a PDF file from the Local Drive into a Google Docs,

Click on the button Convert Local PDF file(s)
The file selection window will be opened
Select one or more PDF file(s) that need to be converted into a Google Docs and Click OK.
The selected PDF files will be converted into Google Docs and the new Google Docs will be opened in a new tab if you have selected it
That's it.

Converting PDF files from Google Drive into Google Docs

To convert a PDF file from the Google Drive into a Google Docs,

Click on the button Convert PDF file(s) from Google Drive
The Google File Picker window will be opened

Select one or more PDF file(s) that need to be converted into a Google Docs and Click OK.
The selected PDF files will be converted into Google Docs and the new Google Docs will be opened in a new tab if you have selected it
That's it.

Bulk converting PDF files into Google Docs

The add-on supports bulk conversion of the PDF files into Google Docs in both the local file conversion and the Google Drive file conversion option. You can select up to a maximum of 25 files in a single conversion.

Privacy and Security

For any file conversion, privacy and security concerns arise for users, particularly when documents may contain confidential data.

This add-on ensures complete privacy and security. The conversion process takes place entirely within your Google account. Your PDF files are not uploaded to any external servers for conversion. Therefore, you can confidently use this add-on for converting any type of confidential documents.

This saves time and effort for you.

Caveats

The maximum number of pages allowed in a PDF file is 75.
Some formatting may not be preserved, often due to differences in the methods used to create the PDF file.
The Google Docs editor add-ons can only be used in the Desktop version and doesn't work with the Google Docs iPhone app or the Android app.

Conclusion

The PDF to Docs editor add-on allows you to convert the PDF file into Google Docs most securely, and privately and keeps the formatting intact.

Convert Google Docs to Markdown – With Git Integration & Image Hosting

Vikram Aruchamy — Fri, 26 Jan 2024 05:40:49 +0000

Google Docs is a free web-based word processor known for its robust and collaborative writing features. But when it comes to publishing the content, Markdown stands out as a good choice.

While Google Docs automatically detect some Markdown formats and converts them into formatted text, it lacks native support for exporting content back into Markdown format. It also doesn't offer functionalities to validate the accessibility structure of documents, such as ensuring proper heading level hierarchy.

In this tutorial, you'll discover how to effortlessly convert Google Docs into Markdown format using the Docs to Markdown Pro Add-on. We'll also explore its powerful features such as seamless GitHub or GitLab integration and robust image handling.

Features at a Glance

Accessibility checks: Checks for missing titles, proper heading levels across and missing image alt text for inclusive and SEO-optimized content

Generate Markdown output: Easily create Markdown output and copy it as text or download it as a file, ready for publishing. If your content includes images, get a publish-ready output as a Zip file containing both Markdown and images

One Click to GitHub/GitLab: Easily push images and Markdown to GitHub or GitLab with one click, streamlining your publishing process and preventing potential errors that might occur during manual uploads

Images optimized for speed: Automatically compresses images before uploading them to Git. This makes web pages load faster and saves space in the Git repository

Static site ready: Automatically generates front matter for Hugo, Jekyll, MkDocs, or Docusaurus when publishing to Git

Blocks Library: Save time and maintain consistency with reusable blocks for inserting boilerplate code and text into your documents

Cloud image hosting for speed: Seamlessly hosts images on the AWS cloud and generates a URL that delivers your images from 600+ AWS CloudFront locations

AI-powered caption generation: Effortlessly creates descriptive image captions using AI

Read further to find out how each feature of Docs to Markdown Pro can enhance your content creation journey.

Accessibility Check Feature

The add-on ensures accessibility from the start of Markdown creation with automatic WCAG checks for title availability, heading levels and image alt text.

Proper heading levels enhance accessibility by structuring content for screen readers, aiding navigation. Additionally, for SEO, they optimize content hierarchy, improving search engine understanding and user experience.

The tool verifies the hierarchy of headings used and flags errors if any headings are incorrectly leveled as shown in the following image:

Image alt text is vital for accessibility, offering descriptions for visually impaired users and improving SEO by providing context of images for search engines.

Additionally, when the internet is slow, the alt text helps readers grasp image content as it loads, making sure they understand what's in the images.

When converting to Markdown, if an image is missing alt text, the add-on shows the image's position and the section heading where the image is present. This makes it easy to add the alt text to that image.

These checks make your content inclusive by ensuring everyone can access it and optimize it for search engines, so more people can find it.

AI-Powered Image Caption Generation

Optionally, the add-on can automatically generate AI captions for images, saving time for the writers.

Download Markdown And Images as Zip

Once your content is converted to Markdown, the add-on creates a Zip file containing both the Markdown file and its images. The Markdown file is updated with the correct image names and paths, making it publish-ready.

You can upload the images in the appropriate location, upload the Markdown file, and hit publish.

Publish to GitHub or GitLab

The add-on seamlessly integrates with both GitHub and GitLab, enabling effortless publishing of your Markdown files onto Git-powered sites. (For example, the GitHub Pages site built with static site generators).

You can push your Google Docs content as Markdown, including images, to GitHub or GitLab with one click. This ensures error-free handling of the image path, image file names and Markdown filename.

This also works for updating previously published Git content.
You can make the necessary updates in Google Docs and click push. The updated content will be pushed as Markdown to Git seamlessly.

Automatic Image Compression

When users include screenshots or images in Docs, these images might take up more space due to their higher quality or resolution.

In this case, the tool automatically compresses these images before uploading to GitHub or GitLab, saving space in Git repositories.

This optimization significantly enhances the website's overall speed and user experience.

Cloud Image Hosting

While converting Docs to Markdown, the tool allows you to upload images to the AWS cloud and generates cloud URLs in your Markdown automatically.

This eliminates manual image uploads to your server and image path updates in Markdown files, ensuring streamlined workflows, and efficient image storage.

There are two main advantages of Cloud image hosting:

Hosting images on the Cloud ensures faster image delivery to users through global edge locations (600+), reducing latency and enhancing accessibility.
Uploading images to Cloud storage instead of Git repositories prevents bloating Git repositories with binary files.

Reusable Blocks

The Blocks library allows you to add boilerplate texts and codes as reusable blocks. You can insert those blocks into your document with a single click.

Advantages of using blocks

Using reusable blocks ensures consistent style in all your content, keeping everything looking uniform and cohesive.

Streamline Your Process with Google Docs to Markdown Pro

Conclusion

The Docs to Markdown Pro Add-on makes Google Docs to Markdown conversion seamless. With accessibility checks, smooth GitHub/GitLab integration, and robust image handling, it streamlines your workflow and ensures content accuracy.

Simplify your workflow and speed up content creation now with the Docs to Markdown Pro Add-on.

How to List Contents of s3 Bucket Using Boto3 Python?

Vikram Aruchamy — Tue, 12 Oct 2021 16:37:48 +0000

S3 is a storage service from AWS. You can store any files such as CSV files or text files. You may need to retrieve the list of files to make some file operations. You'll learn how to list the contents of an S3 bucket in this tutorial.

You can list contents of the S3 Bucket by iterating the dictionary returned from my_bucket.objects.all() method.

If You're in Hurry...

You can use the below code snippet to list the contents of the S3 Bucket using boto3.

Snippet

import boto3

session = boto3.Session( aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')



s3 = session.resource('s3')

my_bucket = s3.Bucket('stackvidhya')

for my_bucket_object in my_bucket.objects.all():
    print(my_bucket_object.key)

Output

    csv_files/
    csv_files/IRIS.csv
    df.csv
    dfdd.csv
    file2_uploaded_by_boto3.txt
    file3_uploaded_by_boto3.txt
    file_uploaded_by_boto3.txt
    filename_by_client_put_object.txt
    text_files/
    text_files/testfile.txt

If You Want to Understand Details, Read on…

In this tutorial, you'll learn the different methods to list contents from an S3 bucket using boto3.

You'll use boto3 resource and boto3 client to list the contents and also use the filtering methods to list specific file types and list files from the specific directory of the S3 Bucket.

Installing Boto3

If you've not installed boto3 yet, you can install it by using the below snippet.

Snippet

%pip install boto3

Boto3 will be installed successfully.

Now, you can use it to access AWS resources.

Using Boto3 Resource

In this section, you'll use the Boto3 resource to list contents from an s3 bucket.

Boto3 resource is a high-level object-oriented API that represents the AWS services. Follow the below steps to list the contents from the S3 Bucket using the Boto3 resource.

Create Boto3 session using boto3.session() method
Create the S3 resource session.resource('s3') snippet
Create bucket object using the resource.Bucket(<Bucket_name>) method.
Invoke the objects.all() method from your bucket and iterate the returned collection to get the each object details and print each object name using thy attribute key.

Note: In addition to listing objects present in the Bucket, it'll also list the sub-directories and the objects inside the sub-directories.

Use the below snippet to list objects of an S3 bucket.

Snippet

import boto3
session = boto3.Session(aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')

#Then use the session to get the resource
s3 = session.resource('s3')

my_bucket = s3.Bucket('stackvidhya')

for my_bucket_object in my_bucket.objects.all():
    print(my_bucket_object.key)

You'll see the list of objects present in the Bucket as below in alphabetical order.

Output

    csv_files/
    csv_files/IRIS.csv
    df.csv
    dfdd.csv
    file2_uploaded_by_boto3.txt
    file3_uploaded_by_boto3.txt
    file_uploaded_by_boto3.txt
    filename_by_client_put_object.txt
    text_files/
    text_files/testfile.txt

This is how you can use the boto3 resource to List objects in S3 Bucket.

Using Boto3 Client

In this section, you'll use the boto3 client to list the contents of an S3 bucket.

Boto3 client is a low-level AWS service class that provides methods to connect and access AWS services similar to the API service. Follow the below steps to list the contents from the S3 Bucket using the boto3 client.

Create Boto3 session using boto3.session() method
Create the boto3 s3 client using the boto3.client('s3') method.
Invoke the list_objects_v2() method with the bucket name to list all the objects in the S3 bucket. It returns the dictionary object with the object details.
Iterate the returned dictionary and display the object names using the obj[key].

Note: Similar to the Boto3 resource methods, the Boto3 client also returns the objects in the sub-directories.

Use the below snippet to list objects of an S3 bucket.

Snippet

import boto3

s3_client = boto3.client('s3', 
                      aws_access_key_id='<your_access_key_id>', 
                      aws_secret_access_key='<your_secret_access_key>' 
                      )

objects = s3_client.list_objects_v2(Bucket='stackvidhya')

for obj in objects['Contents']:
    print(obj['Key'])

You'll see the objects in the S3 Bucket listed below.

Output

    csv_files/
    csv_files/IRIS.csv
    df.csv
    dfdd.csv
    file2_uploaded_by_boto3.txt
    file3_uploaded_by_boto3.txt
    file_uploaded_by_boto3.txt
    filename_by_client_put_object.txt
    text_files/
    text_files/testfile.txt

This is how you can list keys in the S3 Bucket using the boto3 client.

List Contents of A Specific Directory

In this section, you'll learn how to list a subdirectory's contents that are available in an S3 bucket. This will be useful when there are multiple subdirectories available in your S3 Bucket, and you need to know the contents of a specific directory.

You can use the filter() method in bucket objects and use the Prefix attribute to denote the name of the subdirectory.

Filter() and Prefix will also be helpful when you want to select only a specific object from the S3 Bucket.

Use the below snippet to select content from a specific directory called csv_files from the Bucket called stackvidhya.

Snippet

import boto3

session = boto3.Session( aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')

#Then use the session to get the resource
s3 = session.resource('s3')

my_bucket = s3.Bucket('stackvidhya')

for objects in my_bucket.objects.filter(Prefix="csv_files/"):
    print(objects.key)

You'll see the list of objects present in the sub-directory csv_files in alphabetical order.

Output

    csv_files/
    csv_files/IRIS.csv

This is how you can list files in the folder or select objects from a specific directory of an S3 bucket.

List Specific File Types From a Bucket

In this section, you'll learn how to list specific file types from an S3 bucket.

This may be useful when you want to know all the files of a specific type. To achieve this, first, you need to select all objects from the Bucket and check if the object name ends with the particular type. If it ends with your desired type, then you can list the object.

It'll list the files of that specific type from the Bucket and including all subdirectories.

Use the below snippet to list specific file types from an S3 bucket.

Snippet

import boto3

session = boto3.Session( aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')

s3 = session.resource('s3')

my_bucket = s3.Bucket('stackvidhya')

for obj in my_bucket.objects.all():
    if obj.key.endswith('txt'):
        print(obj.key)

You'll see all the text files available in the S3 Bucket in alphabetical order.

Output

    file2_uploaded_by_boto3.txt
    file3_uploaded_by_boto3.txt
    file_uploaded_by_boto3.txt
    filename_by_client_put_object.txt
    text_files/testfile.txt

This is how you can list files of a specific type from an S3 bucket.

List Contents From A directory Using Regular Expression

Boto3 currently doesn't support server side filtering of the objects using regular expressions.

However, you can get all the files using the objects.all() method and filter it using the regular expression in the IF condition.

For example, if you want to list files containing a number in its name, you can use the below snippet. To do an advanced pattern matching search, you can refer to the regex cheat sheet.

Snippet

import re 
import boto3

session = boto3.Session(aws_access_key_id='<your_access_key_id>', aws_secret_access_key='<your_secret_access_key>')

s3 = session.resource('s3')

my_bucket = s3.Bucket('stackvidhya')

substring =  "\d"

for obj in my_bucket.objects.all():
    if re.search(substring,  obj.key):  
        print(obj.key)

You'll see the file names with numbers listed below.

Output

file2_uploaded_by_boto3.txt
file3_uploaded_by_boto3.txt
file_uploaded_by_boto3.txt

This is how you can list contents from a directory of an S3 bucket using the regular expression.

Conclusion

To summarize, you've learned how to list contents for an S3 bucket using boto3 resource and boto3 client. You've also learned to filter the results to list objects from a specific directory and filter results based on a regular expression.

If you have any questions, comment below.

How to Rename Multi index columns in Pandas Dataframe

Vikram Aruchamy — Thu, 05 Aug 2021 10:33:04 +0000

Pandas dataframe is a powerful two dimensional data structure which allows you to store data in rows and columns format.

It also supports multi index for columns and rows.

In this tutorials, you'll learn how to create multi-index pandas data-frame and how to rename the columns of the multi index data-frame.

If you would like to rename columns of a single index dataframe read, How to Rename columns of Pandas Dataframe?

Creating Multi-Index Dataframe

To create a multi index dataframe, you need to follow two steps.

First, create a normal dataframe using pd.DataFrame() method.

Second, set the columns for the dataframe using pd.MultiIndex.from_tuples(). This allows you to set mutli index for the columns of the dataframe.

Snippet

import pandas as pd

df = pd.DataFrame([[1,2,3], [4,5,6], [7,8,9]])

df.columns = pd.MultiIndex.from_tuples((("a", "b"), ("a", "c"), ("a", "d")))

df

You'll see the below output.

Output

    a
    b   c  d
0   1   2   3
1   4   5   6
2   7   8   9

The multilevel column index dataframe is created. a is the first level column index and b, c, d are the second level column indexes.

Next, let's see how to rename these mutli-level columns.

Renaming the Multiindex Columns

To rename the multi index columns of the pandas dataframe, you need to use the set_levels() method.

Use the below snippet to rename the multi level columns.

Snippet

df.columns.set_levels(['b1','c1','d1'],level=1,inplace=True)

df

where,

['b1','c1','d1'] - New column names of the index
level=1 - Level of the columns to be renamed
inplace=True - To perform the rename operation in the same dataframe rather than creating the new dataframe

Now the second level index of the columns will be renamed to b1, c1, d1 as shown below.

Output

    a
    b1  c1  d1
0   1   2   3
1   4   5   6
2   7   8   9

This is how you can rename the columns of the multi index dataframe.

Conclusion

In this short tutorial, you've learnt how to rename the columns of the multi-index pandas data-frame using the set_levels() method.

Pandas Add Row to DataFrame - Definitive Guide

Vikram Aruchamy — Sun, 20 Jun 2021 14:34:07 +0000

Pandas dataframe is a two dimensional data structure. When using the dataframe for data analysis, you may need to create a new dataframe and selectively add rows to dataframe for creating a dataframe with specific records.

You can add rows to pandas dataframe using

append()
concat()
iloc[]
loc[]

If You're in Hurry...

You can use the below code snippet to add row to dataframe.

Snippet

df2 = {'First Name': 'Vikram', 'Last Name': 'Aruchamy', 'Country': 'India'}

df = df.append(df2, ignore_index = True)

df

Dataframe Will Look Like

	Country	First Name	Last Name
0	India	Vikram	Aruchamy

If You Want to Understand Details, Read on…

In this tutorial, you’ll learn the different methods available to add rows to dataframe. You'll also learn how to insert row to an empty dataframe.

Creating an Empty Dataframe

First, you need to create an empty dataframe to add rows to it. You can do it by using DataFrame() method as shown below.

Snippet

import pandas as pd

df = pd.DataFrame()

df

Empty dataframe is created as df.

Add Row to Dataframe

You can add rows to dataframe using four methods. append(), concat(), iloc[] and loc[].

Let's have a look at it one by one.

To create a new row, you need to know the columns already available in the dataframe. Read How to Get Column Name in Pandas to know the columns in the dataframe.

Alternatively, you can print the dataframe using print(df) to know the dataframe columns.

Using Append

You can use the append() method to append a row to an existing dataframe.

Parameters

dictionary or Pandas Series or Dataframe - Object with values for new row
ignore_index = True Means the index from the series or the source dataframe will be ignored. The index available in the target dataframe will be used, instead. False means otherwise. This is optional.

Returns

A resultant dataframe which has the rows from the target dataframe and a new row appended.

inplace append is not possible. Hence, do not forget to assign the result to a dataframe object to access it later.

In the below example, a dictionary is created with values for the columns which already exists in the target dataframe. Then it is appended to the target dataframe using the append() method.

dict = {'First Name': 'Vikram', 'Last Name': 'Aruchamy', 'Country': 'India'}

df = df.append(dict, ignore_index = True)

df

Now, you've appended one row to dataframe.

Dataframe Will Look Like

	Country	First Name	Last Name
0	India	Vikram	Aruchamy

This is how you can insert a row to dataframe using append.

Using Concat

You can append a row to dataframe using concat() method. It concatenates two dataframe into one.

To add one row, create a dataframe with one row and concatenate it to the existing dataframe.

Parameters

It accepts,

List of dataframes - List of dataframes that needs to be concatenated
ignore_index - Whether the index of the new dataframe should be ignored when concatenating to the target dataframe
axis = 0 - To denote that rows of the dataframe needs to be converted. If you want to concatenate columns, you can use axis=1

Returns

It returns a new dataframe object which has the rows concatenated from two dataframes.

inplace concatenation is not supported. Hence, remember to assign the result to a variable for later use.

Snippet

df2 = pd.DataFrame({'First Name': ['Kumar'],
                    'Last Name' : ['Ram'],
                    'Country' : ['India']})

df = pd.concat([df, df2], ignore_index = True, axis = 0)

df

In the above example, you're creating a new dataframe with one rows and it is named as df2. You're concatenating this to dataframe df which already has one dataframe in it.

Both df and df2 will be concatenated and you'll see two rows in the resultant dataframe.

Dataframe Will Look Like

	Country	First Name	Last Name
0	India	Vikram	Aruchamy
1	India	Kumar	Ram

This is how you can use the concat() method to add rows to the dataframe.

Using iLOC

You can use the iLoc[] attribute to add row at a specific position in the dataframe. iloc is an integer based indexing for selecting rows from the dataframe. You can also use it to assign new rows at that position.

Adding a row at a specific index position will replace the existing row at that position.

Note: When you're using iLoc to add a row, the dataframe must already have a row in the position. Atleast an empty row. If a row is not available, you'll see an error IndexError: iloc cannot enlarge its target object. iLoc will not expand the size of the dataframe automatically.

Snippet

df.iloc[1] = ['Shivam', 'Pandey', 'India']

df

In the above example, you're directly adding a row at the index position 1. It replaced the values available in that position with the new values.

Dataframe Will Look Like

	Country	First Name	Last Name
0	India	Vikram	Aruchamy
1	Shivam	Pandey	India

This is how you can use the iloc[] to insert a row to the existing dataframe.

Using LOC

You can add row to dataframe using the loc parameter. loc[] is used to access set of rows from the dataframe using the index label. You can also assign rows with a specific index label using the loc attribute.

When using the loc[] attribute, its not mandatory that a row already exists with specific label. It'll automatically extend the dataframe and add a row with that label, unlike the iloc[] method.

A full program is demonstrated for this method because previous methods has the dataframe with the row indexes 1,2,3..

To demonstrate loc using the row indexes with names like a, b, a new dataframe is created with labels a and b. Then a new row is assigned with the row label c using the loc[] method.

Snippet

import pandas as pd

# List of Tuples
users = [ ('Shivam', 'Pandey', 'India'),
             ('Kumar', 'Ram' , 'India' ),
              ]
#Create a DataFrame object
df3 = pd.DataFrame(  users, 
                    columns = ['First Name' , 'Last Name', 'Country'],
                    index=['a', 'b']) 


print('Dataframe before adding a new row:\n')
print('---------------------------------------\n')
print(df3)

df3.loc['c'] = ['Vikram', 'Aruchamy', 'India']

print('\nDataframe after adding a new row:\n')
print('---------------------------------------\n')

print(df3)

First a dataframe df3 is created with two rows with label a and b. Then a row is inserted with label c using the loc[] method.

Dataframe Will Look Like

    Dataframe before adding a new row:

    ---------------------------------------

      First Name Last Name Country
    a     Shivam    Pandey   India
    b      Kumar       Ram   India

    Dataframe after adding a new row:

    ---------------------------------------

      First Name Last Name Country
    a     Shivam    Pandey   India
    b      Kumar       Ram   India
    c     Vikram  Aruchamy   India

This is how you can use the loc[] method to add rows to the dataframe. Either it is an empty dataframe or it already has values.

Next, you'll see the different circumstances where you can use the loc, iloc, append() or concat() methods to add rows to the dataframe.

Pandas Insert Row at Specific Index

You can insert row at a specific index in a dataframe using the loc method.

This will be useful when you want to insert row between two rows in a dataframe.

Alternatively, you can also use the iloc[] method to add rows at a specific index. However, in that case, there must be a row already existing with specific index.

Points to Note

When using loc[], If a row is already existing with that index label, it'll be replaced with the new row.

Snippet

df.loc[2] = ['Shivam', 'Pandey', 'India']

df

A row will be added with the index label 2.

Dataframe Will Look Like

	Country	First Name	Last Name
0	India	Vikram	Aruchamy
1	Shivam	Pandey	India
2	Shivam	Pandey	India

This is how you can append row at a specific index in a dataframe.

Pandas Insert Row At top

You can insert a row at top in dataframe using the df.loc[-1].

After inserting the row with index -1, you can increment all the indexes by 1.

Now indexes of the rows in the dataframe will be 0,1,2,..n-1.

Note

To use this method, the index labels of the rows must be integers. Otherwise, it won't work.

Snippet

df.loc[-1] = ['Raj', 'Kumar', 'India']

df.index = df.index + 1

df = df.sort_index()

df

Row is first added at position -1 and then all the indexes will be incremented and sorted.

Dataframe Will Look Like

	Country	First Name	Last Name
0	Raj	Kumar	India
1	India	Vikram	Aruchamy
2	Shivam	Pandey	India
3	Shivam	Pandey	India

This is how you can insert row at top of the dataframe.

Pandas Insert Row at Bottom

You can insert a row at the bottom in dataframe using the df.loc[df.shape[0]].

df.shape[0] returns the length of the dataframe.

For example, if a dataframe already contains 3 rows, already existing rows will have the index 0,1,2,3. Shape[] method will return 4. Hence when you insert using loc[4], a row will be added at bottom of the dataframe which is index 4.

Snippet

df.loc[df.shape[0]] = ['Krishna', 'Kumar', 'India']

df

A new row will be added at the index position 4 as you see below.

Dataframe Will Look Like

	Country	First Name	Last Name
0	Raj	Kumar	India
1	India	Vikram	Aruchamy
2	Shivam	Pandey	India
3	Shivam	Pandey	India
4	Krishna	Kumar	India

This is how you can append a row at the bottom of the dataframe using loc[].

Pandas Insert Empty Row

You may need append an empty row to pandas dataframe for adding a row to it later. You can also fill value for specific columns in the dataframe after creating an empty row.

Empty rows can be appended by using the df.loc[df.shape[0]] and assigning None values for all the existing columns.

For example, if your dataframe has three columns, you can create a series with 3 None values and assign it at the last position of the dataframe.

That is how you can insert an empty row to the dataframe.

Snippet

df.loc[df.shape[0]] = [None, None, None]

df

An empty row is added at the end of the dataframe.

Dataframe Will Look Like

	Country	First Name	Last Name
0	Raj	Kumar	India
1	India	Vikram	Aruchamy
2	Shivam	Pandey	India
3	Shivam	Pandey	India
4	Krishna	Kumar	India
5	None	None	None

This is how you can add empty row to the end of the dataframe.

Pandas Append Two Dataframe Pandas

You can append a dataframe to another dataframe using the dataframe append() method.

append() method accepts a dataframe and appends it to the calling dataframe and returns a new dataframe object.

inplace append is not possible. hence you need to assign the result a dataframe object if you want to use it later.

ignore_index can be used to ignore the index of the dataframe that is assigned to the target dataframe.

Snippet

df2 = {'First Name': 'Vikram', 'Last Name': 'Aruchamy', 'Country': 'India'}

df = df.append(df2, ignore_index = True)

df

In the above example, dataframe df2 is appended to df and assigned it back to the df object.

Dataframe Will Look Like

	Country	First Name	Last Name
0	Raj	Kumar	India
1	India	Vikram	Aruchamy
2	Shivam	Pandey	India
3	Shivam	Pandey	India
4	Krishna	Kumar	India
5	None	None	None
6	India	Vikram	Aruchamy

This is how you can append two dataframe in pandas using the append() method.

Why You Should Not Add Rows One By One To Dataframe

You may need to create a dataframe and append one row at a time in various scenarios.

In that case, it is advisable to create a list first to hold all the records and create a dataframe with all the records in one shot using the pd.DataFrame()method.

Calling the append() method for each row is a costlier operation. But adding the rows to the list is not costlier. Hence, you can add to list and create a dataframe using that list.

Snippet

data = []

data.append(['Krishna', 'Kumar', 'India'])

data.append(['Ram', 'Kumar', 'India'])

data.append(['Shivam', 'Pandey', 'India'])

df = pd.DataFrame(data, columns=['First Name', 'Last Name', 'Country'])

df

For more details about this scenario, refer stackoverflow answer.

Dataframe Will Look Like

	First Name	Last Name	Country
0	Krishna	Kumar	India
1	Ram	Kumar	India
2	Shivam	Pandey	India

This is how you can create pandas dataframe by appending one row at a time.

Conclusion

To summarize, you've learnt how to create empty dataframe in pandas and add rows to it using the append(), iloc[], loc[], concatenating two dataframes using concat().

Also, how these methods can be used to insert row at specific index, add row to the top or bottom of the dataframe, how to add an empty row to the dataframe which can be used at later point.

In addition to that, you've learnt why you should not create pandas dataframe by appending one row at a time and use a list in such scenarios and create a dataframe using the list.

If you have any questions, comment below.

How to write a file or data to an S3 object using boto3

Vikram Aruchamy — Sat, 19 Jun 2021 08:19:59 +0000

S3 is an object storage service proved by AWS. You may need to upload data or file to S3 when working with AWS Sagemaker notebook or a normal jupyter notebook in Python.

You can Write a file or data into S3 Using Boto3 using,

Object.put()
Upload_File()
Client.putObject()

If You're in Hurry...

You can use the below code snippet to write a file to S3.

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

txt_data = b'This is the content of the file uploaded from python boto3 asdfasdf'

object = s3.Object('<bucket_name>', 'file_uploaded_by_boto3.txt')

result = object.put(Body=txt_data)

This is how you can upload file to S3 from Jupyter notebook and Python using Boto3.

If You Want to Understand Details, Read on…

In this tutorial, you'll learn how to write a file or a data to S3 using Boto3.

Prerequisites

Generate the security credentials by clicking Your Profile Name -> My security Credentials -> Access keys (access key ID and secret access key) option. This is necessary to create session to your S3 bucket.
Understand the difference between boto3 resource and boto3 client. Object.put() and the upload_file() methods are from boto3 resource where as put_object() is from boto3 client.

Using Object.put()

You can use the Object.put() method available in the S3 object.

It allows two ways to write into the S3.

Writing a text content to an S3 object
Writing contents from the local file to the S3 object

Write Text Data To S3 Object

In this section, you'll learn how to write a normal text data to the s3 object.

Follow the below steps to write a text data to an S3 Object.

Create a Boto3 session using the security credentials
With the session, create a resource object for the S3 service
Create an S3 object using the object method. It accepts two parameters. BucketName and the File_Key. File_Key is the name you want to give it for the S3 object. If you would like to create sub-folders inside the bucket, you can prefix the locations in this File_key variable. For example, /subfolder/file_name.txt
1. Create an text object which holds the text to be updated to the S3 object
2. Use the put() action available in the S3 object and the set the body as the text data. E.g. Body=txt_data.
3. put() actions returns a JSON response metadata. This metadata contains the HttpStatusCode which shows if the file upload is successful or not. If the status code is 200, then the file upload is successful. Else, it is not.

Note: Using this method will replace the existing S3 object in the same name. Hence ensure you're using a unique name to this object.

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

object = s3.Object('<bucket_name>', 'file_uploaded_by_boto3.txt')

txt_data = b'This is the content of the file uploaded from python boto3'

result = object.put(Body=txt_data)

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded Successfully')
else:
    print('File Not Uploaded')

Output

    File Uploaded Successfully

This is how you can update the text data to an S3 object using Boto3.

Reading a File from Local and Updating it to S3

In this section, you'll learn how to read a file from local system and update it to an S3 object.

It is similar to the steps explained in the previous step except for one step.

You just need to open a file in the binary mode and send its content to the put() method using the below snippet.

Use only forward slash for the filepath. Backslash doesn't work.

Note: Using this method will replace the existing S3 object in the same name. Hence ensure you're using a unique name to this object.

result = object.put(Body=open('E:/temp/testfile.txt', 'rb'))

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

object = s3.Object('<bucket_name>', 'file3_uploaded_by_boto3.txt')

result = object.put(Body=open('E:/temp/testfile.txt', 'rb'))

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded Successfully')
else:
    print('File Not Uploaded')

You can check if file is successfully uploaded or not using the HTTPStatusCode available in the responsemetadata.

Output

    File Uploaded Successfully

This is how you can write the data from the text file to an S3 object using Boto3.

Using Upload_File()

In this section, you'll learn how to use the upload_file() method to upload a file to an S3 bucket. It is a boto3 resource.

Follow the below steps to use the upload_file() action to upload file to S3 bucket.

Create a boto3 session
Create an object for S3 object
Access the bucket in the S3 resource using the s3.Bucket() method and invoke the upload_file() method to upload the files
upload_file() method accepts two parameters.

- `File_Path` - Path of the file from the local system that needs to be uploaded. Use only forward slash when you mention the path name
- `Object_name` - Name for the object that will be created by uploading this file

Unlike the other methods, the upload_file() method doesn't return an meta object to check the result. You can use the other methods to check if an object is available in the bucket.

Note: Using this method will replace the existing S3 object in the same name. Hence ensure you're using a unique name to this object.

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

result = s3.Bucket('<bucket_name>').upload_file('E:/temp/testfile.txt','file2_uploaded_by_boto3.txt')

print(result)

File is updated successfully. But you'll only see the status as None.

Output

    None

This is how you can use the upload_file() method to upload file to the S3 buckets.

Using Client.putObject()

In this section, you'll learn how to use the put_object method from the boto3 client.

Follow the below steps to use the client.put_object() method to upload a file as an S3 object.

Create a boto3 session using your AWS security credentials
Create a resource object for S3
Get the client from the S3 resource using s3.meta.client
Invoke the put_object() method from the client. It accepts two parameters.

 - `body`  - To pass the textual content for the S3 object. You can pass the text directly. Or you can use the file object by opening the file using `open('E:/temp/testfile.txt', 'rb')`
 - `Name` - Name for the new object that will be created.

put_object() also returns a ResponseMetaData which will let you know the status code to denote if the upload is successful or not.

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

result = s3.meta.client.put_object(Body='Text Contents', Bucket='<bucket_name>', Key='filename_by_client_put_object.txt')

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded Succesfully')
else:
    print('File Not Uploaded')

A new S3 object will be created and the contents of the file will be uploaded.

Output

    File Uploaded Succesfully

This is how you can use the put_object() method available in boto3 S3 client to upload files to the S3 bucket.

Conclusion

To summarize, you've learnt what is boto3 client and boto3 resource in the prerequisites and also learnt the different methods available in the boto3 resource and boto3 client to upload file or data to the S3 buckets.

How To Load Data From AWS S3 into Sagemaker (Using Boto3 or AWSWrangler)

Vikram Aruchamy — Fri, 11 Jun 2021 16:39:34 +0000

How To Load Data From AWS S3 into Sagemaker (Using Boto3 or AWSWrangler)

S3 is a storage service from AWS. You can store any type of files such as csv files or text files. SageMaker provides the compute capacity to build, train and deploy ML models.

You can load data from AWS S3 into AWS SageMaker using Boto3 or AWSWranger.

In this tutorial, you'll learn how to load data from AWS S3 into SageMaker jupyter notebook.

This will only access the data from S3. The files will not be downloaded to the SageMaker Instance itself. If you want to download the file to the SageMaker policy, read How to Download File From S3 Using Boto3 [Python]?

Prerequisite

Sagemaker instance MUST have read access to your S3 buckets. Assign the role AmazonSageMakerServiceCatalogProductsUseRole while creating SageMaker instance. Refer this link for more details about SageMaker Roles
Install pandas dataframe using pip install pandas to read csv file as dataframe. In most cases it is available as default package

Loading CSV file from S3 Bucket Using URI

In this section, you'll load the CSV file from the S3 bucket using the S3 URI.

There are two options to generate the S3 URI

Copying object URL from the AWS S3 Console.
Generate the URI manually by using the String format option. (This is demonstrated in the below example)

Follow the below steps to load the CSV file from S3 bucket.

Import pandas package to read csv file as a dataframe
Create a variable bucket to hold the bucket name.
Create the file_key to hold the name of the s3 object. You can prefix the subfolder names, if your object is under any subfolder of the bucket.
Concatenate the bucket name and the object name with the prefix s3:// to generate the URI of the S3 object
Use the generated URI in the read_csv() method of the pandas package and store it in the dataframe object called df

In the example, the object is available in the bucket stackvidhya and sub-folder called csv_files. Hence you'll use the bucket name as stackvidhya and the file_key as csv_files/IRIS.csv

Snippet

import pandas as pd

bucket='stackvidhya'

file_key = 'csv_files/IRIS.csv'

s3uri = 's3://{}/{}'.format(bucket, file_key)

df = pd.read_csv(s3uri)

df.head()

The csv file will be read from the S3 location as a pandas dataframe.

You can print the dataframe using df.head() which will print the first five rows of the dataframe as shown below.

Dataframe will look like

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	sepal_length	sepal_width	petal_length	petal_width	species
0	5.1	3.5	1.4	0.2	Iris-setosa
1	4.9	3.0	1.4	0.2	Iris-setosa
2	4.7	3.2	1.3	0.2	Iris-setosa
3	4.6	3.1	1.5	0.2	Iris-setosa
4	5.0	3.6	1.4	0.2	Iris-setosa

This is how you can load s3 data into sagemaker jupyter notebook without using any external libraries.

In this method, the file is also not downloaded into notebook directly.

Next, you'll learn about using the external libraries to load the data.

Loading CSV file from S3 Bucket using Boto3

In this section, you'll use the Boto3.

Boto3 is an AWS SDK for creating, managing and access AWS services such as S3 and EC2 instances.

Follow the below steps to access the file from S3

Import pandas package to read csv file as a dataframe
Create a variable bucket to hold the bucket name.
Create the file_key to hold the name of the s3 object. You can prefix the subfolder names, if your object is under any subfolder of the bucket.
Create an s3 client using the boto3.client('s3'). Boto3 Client is a low level representation of the AWS services.
Get s3 object using the s3_client.get_object() method. Pass the bucket name and the file key you can created in the previous step. It'll return the s3 data as response. (Stored as obj)
Read the the object body using obj['Body'].read(). It'll return the bytes. Convert these bytes to String using io.BytesIO().
This string can be passed to read_csv() available in pandas. Then you'll get a dataframe.

Snippet

import pandas as pd
import boto3
import io

bucket='stackvidhya'

file_key = 'csv_files/IRIS.csv'

s3_client = boto3.client('s3')

obj = s3_client.get_object(Bucket=bucket, Key=file_key)

df = pd.read_csv(io.BytesIO(obj['Body'].read()))

df.head()

The dataframe can be printed using the df.head() method. It'll print the first five rows of the dataframe as shown below.

Dataframe Will Look Like

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	sepal_length	sepal_width	petal_length	petal_width	species
0	5.1	3.5	1.4	0.2	Iris-setosa
1	4.9	3.0	1.4	0.2	Iris-setosa
2	4.7	3.2	1.3	0.2	Iris-setosa
3	4.6	3.1	1.5	0.2	Iris-setosa
4	5.0	3.6	1.4	0.2	Iris-setosa

You can also use the same steps to access files from S3 in jupyter notebook*(outside of sagemaker)*.

Just pass the AWS API security credentials while creating boto3 client as shown below. Refer the tutorial How to create AWS security credentials to create credentials.

Snippet

s3_client = boto3.client('s3', aws_access_key_id='AWS_SERVER_PUBLIC_KEY', aws_secret_access_key='AWS_SERVER_SECRET_KEY', region_name=REGION_NAME )

This is how you can read csv file into sagemaker using boto3.

Next, you'll learn about the package awswrangler.

Loading CSV File into Sagemaker using AWS Wrangler

In this section, you'll learn how to access data from AWS s3 using AWS Wrangler.

AWS Wrangler is an aws professional service open source python library that extends the functionalities of the pandas library to AWS by connecting dataframe and other data related services.

This package is not installed by default.

Installing AWSWrangler

Install the awswrangler by using the pip install command.

% needs to be prefixed to pip command so the installation directly works from the jupyter notebook.

Snippet

%pip install awswrangler

You'll see the below messages and the AWS Data wrangler will be installed.

Output

    Collecting awswrangler
      Downloading awswrangler-2.8.0-py3-none-any.whl (179 kB)

    Installing collected packages: scramp, redshift-connector, pymysql, pg8000, awswrangler
    Successfully installed awswrangler-2.8.0 pg8000-1.19.5 pymysql-1.0.2 redshift-connector-2.0.881 scramp-1.4.0
    Note: you may need to restart the kernel to use updated packages.

Now, restart the kernel using the Kernel -> Restart option for activating the package.

Once the kernel is restarted, you can use the awswrangler to access data from aws s3 in your sagemaker notebook.

Follow the below steps to access the file from S3 using AWSWrangler.

import pandas package to read csv file as a dataframe
import awswrangler as wr
Create a variable bucket to hold the bucket name.
Create the file_key to hold the name of the S3 object. You can prefix the subfolder names, if your object is under any subfolder of the bucket.
Concatenate bucket name and the file key to generate the s3uri.
1. Use the read_csv() method in awswrangler to fetch the S3 data using the line wr.s3.read_csv(path=s3uri).

Snippet

import awswrangler as wr

import pandas as pd

bucket='stackvidhya'

file_key = 'csv_files/IRIS.csv'

s3uri = 's3://{}/{}'.format(bucket, file_key)

df = wr.s3.read_csv(path=s3uri)

df.head()

readcsv() method will return a pandas dataframe out of csv data. You can print the dataframe using df.head() which will return the first five rows of the dataframe as shown below.

Dataframe Will Look Like

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	sepal_length	sepal_width	petal_length	petal_width	species
0	5.1	3.5	1.4	0.2	Iris-setosa
1	4.9	3.0	1.4	0.2	Iris-setosa
2	4.7	3.2	1.3	0.2	Iris-setosa
3	4.6	3.1	1.5	0.2	Iris-setosa
4	5.0	3.6	1.4	0.2	Iris-setosa

This is how you can load the csv file from S3 using awswrangler.

Next, you'll see how to read a normal text file.

Read Text File from S3

You've seen how to read the csv file from S3 in a sagemaker notebook.

In this section, you'll see how to access a normal text file from S3 and read its content.

As seen before, you can create an S3 client and get the object from S3 client using the bucket name and the object key.

Then you can read the object body using the read() method.

The read method will return the file contents as bytes.

You can decode the bytes into string using the contents.decode('utf-8'). UTF-8 is the most use charset encoding.

Snippet

import boto3

bucket='stackvidhya'

data_key = 'text_files/testfile.txt'

s3_client = boto3.client('s3')

obj = s3_client.get_object(Bucket=bucket, Key=data_key)

contents = obj['Body'].read()

print(contents.decode("utf-8"))

Output

This is a test file to demonstrate the file access functionlity from AWS S3 into sagemaker notebook

Conclusion

To summarize, you've learnt how to access or load the file from aws S3 into sagemaker jupyter notebook using the packages boto3 and awswrangler.

You've also learnt how to access the file without using any additional packages.

If you've any questions, feel free to comment below.

Forem: Vikram Aruchamy

[Boost]

ChatGPT to Google Doc

ChatGPT to Google Doc

Installating the Extension

Installing on Google Chrome

Installing on Microsoft Edge

Refresh ChatGPT after Installing Extension

Exporting Single Response From ChatGPT into Google Doc

Exporting Entire ChatGPT Conversation to Google Doc

Option 1: Exporting entire Conversation with User Prompts

Option 2: Export Only ChatGPT Responses (Excluding User Prompts)

Exporting ChatGPT Canvas Content into a Google Doc

Problems Solved by the ChatGPT to Google Docs or PDF Chrome Extension

Conclusion

Export Google Docs to PDF, But Smarter: Merge, Protect, and Export with Ease

Limitations of Native Google Docs PDF Export

Introducing Docs to PDF Pro

How to Export Google Docs to PDF Using Docs to PDF Pro

Installation Steps

Features Explained

Conclusion

How to Convert Markdown to Google Docs

Installing Docs to Markdown Pro

Converting Markdown to Google Docs using Docs to Markdown Pro

Docs to Markdown Pro - Other Features

Some Usecases of the Markdown to Google Docs

Can Google Docs open a Markdown file?

Wordable Alternative

Docs to WP Pro vs Wordable

Features of Docs to WP Pro - A Wordable Alternative

How to Convert PDF File into a Google Docs

Installing the PDF to Docs Add-on

Add on Permissions

PDF to Docs Add on Features

Converting PDF files from Local Drive into a Google Docs

Converting PDF files from Google Drive into Google Docs

Bulk converting PDF files into Google Docs

Privacy and Security

Caveats

Conclusion

Convert Google Docs to Markdown – With Git Integration & Image Hosting

Features at a Glance

Accessibility Check Feature

AI-Powered Image Caption Generation

Download Markdown And Images as Zip

Publish to GitHub or GitLab

Automatic Image Compression

Cloud Image Hosting

Reusable Blocks

Advantages of using blocks

Streamline Your Process with Google Docs to Markdown Pro

Conclusion

How to List Contents of s3 Bucket Using Boto3 Python?

Installing Boto3

Using Boto3 Resource

Using Boto3 Client

List Contents of A Specific Directory

List Specific File Types From a Bucket

List Contents From A directory Using Regular Expression

Conclusion

You May Also Like

How to Rename Multi index columns in Pandas Dataframe

Creating Multi-Index Dataframe

Renaming the Multiindex Columns

Conclusion

Pandas Add Row to DataFrame - Definitive Guide

Creating an Empty Dataframe

Add Row to Dataframe

Using Append

Using Concat

Using iLOC

Using LOC

Pandas Insert Row at Specific Index

Pandas Insert Row At top

Pandas Insert Row at Bottom

Pandas Insert Empty Row

Pandas Append Two Dataframe Pandas

Why You Should Not Add Rows One By One To Dataframe

Conclusion