Forem: Mihai Chiorean

From Mobile Simplicity to Pi Complexity

Mihai Chiorean — Tue, 15 Oct 2024 18:24:39 +0000

As a backend developer with a solid grounding in Linux and a fair share of experience in mobile development for both Android and iOS, I recently decided to venture into the world of Raspberry Pi 5. I naively assumed that setting up a Raspberry Pi would be as straightforward as plugging in an Android device or an iPhone and hitting the run button in my IDE. Spoiler alert: it wasn’t.

The Illusion of Plug and Play

In mobile development, we’re spoiled by the simplicity of connecting a device via USB and seamlessly deploying our code. The development environments are mature, and the tooling is designed to minimize friction. With Raspberry Pi, I quickly realized that this convenience doesn’t translate.

The OS Conundrum

The first hurdle was choosing an operating system and flashing it onto an SD card. Unlike mobile devices that come with a pre-installed OS, the Raspberry Pi requires you to select and install one yourself. Options like Raspberry Pi OS, Ubuntu, and others each come with their own quirks and compatibility considerations. This step isn’t just a minor setup task; the choice of OS affects not only performance but also the tools and libraries available to you and it’s a critical decision that affects the entire development experience.

The Peripheral Predicament

Most online resources assume you’re plugging in peripherals like a mouse, keyboard, and monitor directly into the Raspberry Pi and doing development on the device itself. However, this approach can be impractical for developers who might not have these peripherals readily available. Instead, we often have powerful laptops with our own development environments already set up — complete with local dotfiles, custom IDE configurations, and personalized editor settings. Recreating all of this on a single-board computer feels counterintuitive when we could leverage our existing setups.

The USB-C Connectivity Challenge

Attempting to connect the Raspberry Pi 5 to my MacBook Pro via USB-C didn’t yield the immediate connection I expected. There was no automatic recognition, no prompt to trust the device, nothing. It turns out that setting up USB networking with the Raspberry Pi requires additional configuration. I stumbled upon this guide, which, while helpful, was far from trivial. It involved modifying configuration files and setting up network interfaces manually.

SSH: A Necessary Evil?

Some developers might not have peripherals like a mouse, keyboard, and monitor readily available, making SSH the default method of interaction despite its learning curve.

Once the OS is installed, connecting to the Raspberry Pi becomes the next challenge. For many, SSH emerges as the primary means of interaction. However, not everyone has extensive experience with SSH, and setting it up can introduce additional complexities.

Relying on SSH adds another layer of detachment from the development process. Instead of the immediate feedback loop we’re accustomed to, every code change requires building and deploying over an SSH connection — often without the safety net of an interactive debugger. This can significantly disrupt the development workflow, especially when compared to the seamless experience of mobile development environments.

The IDE and Development Stack Dilemma

In mobile development, IDEs like Xcode and Android Studio provide a rich set of tools, including debugging, breakpoints, and seamless deployment. With the Raspberry Pi, there’s no out-of-the-box IDE that offers this level of integration. You’re faced with choosing a development stack that may or may not support all the features you need. Do you develop directly on the Pi, or do you set up a cross-compilation environment? Each option has its own set of complexities.

The Debugging Desert

Speaking of debugging, the lack of straightforward debugging tools was perhaps the most jarring difference. Setting breakpoints and inspecting variables in real-time is a staple of my usual workflow. With the Raspberry Pi, setting up such an environment requires significant effort, and in some cases, might not even be feasible depending on the chosen development stack.

Breaking Through: Achieving a Streamlined Setup

Despite the initial challenges, I eventually succeeded in setting up the Raspberry Pi to be recognized by macOS. After configuring the necessary USB networking settings and tweaking some configurations, my MacBook Pro could finally communicate with the Raspberry Pi directly. This breakthrough was significant — it meant I could leverage my existing development environment without the need for additional peripherals.

But I didn’t stop there. I went the extra mile and created a custom OS image with all my configurations, dependencies, and settings pre-installed. This image could be flashed onto another Raspberry Pi and would work out of the box. The ability to replicate the setup effortlessly on multiple devices not only saved time but also opened up possibilities for scaling projects or collaborating with others who could benefit from a ready-to-use environment.

To enhance the development experience further, I’m currently working on integrating Visual Studio Code into my workflow. By utilizing VSCode’s remote development capabilities — or possibly developing a custom extension — I aim to replicate the rich IDE features I’m accustomed to. This setup would allow me to edit, build, and debug code on the Raspberry Pi directly from VSCode on my MacBook Pro. It brings back the convenience of a seamless development environment, bridging the gap between the powerful tools on my laptop and the Raspberry Pi’s hardware capabilities.

Conclusion

While the journey to get a simple ‘Hello World’ running on the Raspberry Pi 5 was unexpectedly convoluted, breaking through those barriers proved incredibly rewarding. By configuring the device to work seamlessly with my MacBook Pro and leveraging tools like VSCode, I’ve started to bridge the gap between powerful development environments and the Raspberry Pi’s unique capabilities. Yes, the initial setup is complex and can be a barrier to entry, but with persistence and the right resources, it’s a challenge that can be overcome. For developers willing to venture into this space, the possibilities are vast and well worth the effort.

Invitation to Connect

If you’re interested in trying out this setup or if you have your own experiences with Raspberry Pi development, I’d love to hear from you! Please feel free to reach out through the comments section below or connect with me on social media. Let’s collaborate and share our insights to make the Raspberry Pi development experience better for everyone.

References:

https://blog.hardill.me.uk/2023/01/07/raspberry-pi-usb-gadget-creator-update/

This is cross posted from https://medium.com/@mihaichiorean/from-mobile-simplicity-to-pi-complexity-3d18d8916fcc

Querying CSV files in AWS S3 from Go, using SQL

Mihai Chiorean — Fri, 10 Jul 2020 16:45:06 +0000

tl;dr

S3 Select allows you to query CSV data in a CSV/JSON file stored in S3 using SQL queries.

I'm going to present an example of how I did this in Go, since I had a hard time finding clear examples while trying to work on it.

Update: now with command line tool

Benefits

only the selected records need to come over the wire, not the whole file
no need to keep some datastore in sync with the latest csv to query the information
reduction in code complexity on the consumer end
can arguably be done sync, just like calling any other external API **

S3 Select in Golang

In this example I used a simple CSV I found online as an example.
I've structured it how I would in an application: a package that wraps the s3 specific code and provides some interface.

.
├── FL_insurance_sample.csv
├── README.md
├── client
│   └── csvdb
│       ├── client.go
│       └── client_test.go
├── config.yaml
├── go.mod
├── go.sum
├── main.go
└── vendor

First, the imports:

 3 import (
       ....
 7
 8     "github.com/aws/aws-sdk-go/aws"
 9     "github.com/aws/aws-sdk-go/service/s3"
10     "github.com/aws/aws-sdk-go/service/s3/s3iface"
       ....
12 )

You'll notice this uses the first version of the Go AWS SDK. There is a v2, however, at the time of this writing it was still v0.x.x.

Next, I put the client struct together. It's fairly basic - wraps the S3 API interface and the bucket + key of the resource it will access.

32 // Client represents the struct used to make queries to an s3 csv
33 type Client struct {
34     s3iface.S3API
35     // S3 path - prefix + file/resource name
36     key    string
37     bucket string
38 }
39
40 // NewClient instantiates a new client struct
41 func NewClient(s s3iface.S3API, bucket, key string) *Client {
42     return &Client{
43         S3API:  s,
44         bucket: bucket,
45         key:    key,
46     }
47 }

Now let's move on to the interesting bits. First, we need to create the input for the S3 query

65     req := &s3.SelectObjectContentInput{
66         Bucket:         aws.String(c.bucket),
67         Key:            aws.String(c.key),
68         Expression:     aws.String(q),
69         ExpressionType: aws.String("SQL"),
70         InputSerialization: &s3.InputSerialization{
71             CSV: &s3.CSVInput{
72                 // query using header names. This is a choice for this example
73                 // many csv files do not have a header row; In that case,
74                 // this property would not be needed and the "filters" would be
75                 // on the column index (e.g. _1, _2, _3...)
76                 FileHeaderInfo: aws.String("Use"),
77             },
78         },
79     }

A few things here:

Expression: is the actual SQL query which looks something like SELECT * FROM s3object s WHERE s.column = 'value' (Read more here
InputSerialization can be CSV or JSON. I'm focusing on CSV for this example.
FileHeaderInfo should be set to "Use" if you plan on using column names (header) in your CSV. Otherwise you'd have to use s._1, s._2 in your query - address the column by it's index.
the AWS Go SDK uses *string a lot. They've made a convenience function to deal with that. Hence the use of aws.String(...)

Next, let's set up the response serialization for this request:

82     req = req.SetOutputSerialization(&s3.OutputSerialization{
83         JSON: &s3.JSONOutput{},
84     })

Similar to the input, both CSV and JSON can be used. I've used json here because it's easy to parse and populate a struct in Go. I've not done csv parsing in Go to be honest.

Ready to call S3 now.

85     out, err := c.SelectObjectContentWithContext(ctx, req)
86     if err != nil {
87         return nil, err
88     }
89
90     stream := out.GetEventStream()
91     defer stream.Close()

The output of this call exposes a stream of events that we need to read the resulting data. This stream is implemented using a channel and sends some extra data apart from the actual rows themselves. The types of events exposed are:

ContinuationEvent - not sure what this is
EndEvent - indicates no more messages will be sent; request is completed
ProgressEvent - data about the progress of an operation
RecordsEvent - the actual rows
StatsEvent - stats about the operation: total bytes, bytes processed etc

Testing considerations

When unit testing this example, I ran into some issues.
My intuition told me I should create a mock of s3iface.S3API and mock the response from the call to SelectObjectContentWithContext. However this didn't work as expected.

The s3.SelectObjectContentOutput contains a SelectObjectContentEventStream which does not have a factory method that I could find. It has some private fields. Instantiating it when mocking proved to be problematic because those private fields are nil and are used in the calls .Err() and .Close().

The route I took is to use net/http/httptest to "mock" the S3 backend instead. I created an http test server with a generic handler and pointed the AWS/S3 session make requests to this.

Update

Once finishing this post I realized that it might be useful to have an easy way to use this example to try queries on a specific CSV. The repo now has a main and you can build a command line tool to run some queries.

Install

git clone https://github.com/mihai-chiorean/s3-select-example.git && cd s3-select-example/cmd/s3ql

go install

Usage

~/w/s3-select-example ❯❯❯ s3ql --help
A small cli to run/test sql queries against a CSV in AWS S3

Usage:
  s3ql [flags]

Flags:
  -b, --bucket string   S3 bucket where the data resides
  -h, --help            help for s3ql
  -k, --key string      The S3 resource - prefix/key - of the CSV file
  -r, --region string   The AWS region where the bucket is
  -t, --use-headers     Tells S3 Select that the csv has a header row and to use it in the query. (default true)

s3ql --bucket <bucket> --key FL_insurance_sample.csv --region us-east-2 select \* from s3object s where s.statecode = \'FL\'

Will print out, line by line, the json output of each matched record in the CSV.