DEV Community

Uri Shaked
Uri Shaked

Posted on

1

Exporting Google Cloud Storage File List to BigQuery (CSV)

Sometimes you need to run complex queries against the list of files storage in your Google Cloud Storage. In my case, I wanted find out which files were no longer referenced by my database and clean them up.

Here is how I did it:

Exporting the File List to a CSV file

This simple 1-line script will allow you to the entire file list from Google Cloud Storage to a simple CSV format that can be then loaded into Google BigQuery:

gsutil ls -l gs://bucket-name/** | head -n-1 | awk 'BEGIN{print "Size,Modified,Path"}{print $1","$2",\""$3"\""}' > filelist.csv

Make sure to replace bucket-name with your actual GCS bucket name.

Loading the CSV file to BigQuery

Before you can load the CSV File to Google's BigQuery, you first need to upload it to Google Cloud storage. You can do it by running:

gsutil cp filelist.csv gs://some-bucket/filelist.csv

Then, you can load it right into bigquery:

bq load --autodetect --source_format=CSV mydataset.mytable gs://some-bucket/filelist.csv

Make sure you change some-bucket to your actual bucket name, and mydataset and mytable to the target BigQuery dataset and table, respectively.

I tested this process for a bucket containing about 350,000 files and it worked flawlessly.

Redis image

Short-term memory for faster
AI agents

AI agents struggle with latency and context switching. Redis fixes it with a fast, in-memory layer for short-term context—plus native support for vectors and semi-structured data to keep real-time workflows on track.

Start building

Top comments (0)

Dev Diairies image

User Feedback & The Pivot That Saved The Project

🔥 Check out Episode 3 of Dev Diairies, following a successful Hackathon project turned startup.

Watch full video 🎥

👋 Kindness is contagious

Discover fresh viewpoints in this insightful post, supported by our vibrant DEV Community. Every developer’s experience matters—add your thoughts and help us grow together.

A simple “thank you” can uplift the author and spark new discussions—leave yours below!

On DEV, knowledge-sharing connects us and drives innovation. Found this useful? A quick note of appreciation makes a real impact.

Okay