Forem: Suyash K.

Django Test Driven Social Media REST API - Getting Started

Suyash K. — Wed, 08 Feb 2023 11:57:03 +0000

Basic Project Setup

mkdir DjangoRESTAPI && cd $_
python3 -m venv .venv
. .venv/bin/activate
pip install django djangorestframework
pip freeze > requirements.txt
git init
curl https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore > .gitignore
django-admin startproject config .

This should set up a boilerplate project with project configs and initialized git repository.
The Project Structure should resemble the following.

$ tree -a --gitignore -I ".git/"
.
├── .gitignore
├── config
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
└── requirements.txt

$ cat requirements.txt 
asgiref==3.6.0
backports.zoneinfo==0.2.1
Django==4.1.6
djangorestframework==3.14.0
pytz==2022.7.1
sqlparse==0.4.3

Initial Commit: Getting Started

mkdir -p src/user
touch src/__init__.py
python3 manage.py startapp user src/user

The Project Structure should resemble the following:

$ tree src/
src/
├── __init__.py
└── user
    ├── __init__.py
    ├── admin.py
    ├── apps.py
    ├── migrations
    │   └── __init__.py
    ├── models.py
    ├── tests.py
    └── views.py

Now, to add the user app to the project, edit src/user/apps.py

from django.apps import AppConfig

class UserConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'src.user'

and then edit config/settings.py

# ...

INSTALLED_APPS = [
    'django.contrib.admin',
    # ...
    'src.user'
]
# ...

Now we can run migrations and start the development server.

python3 manage.py makemigrations
python3 manage.py migrate
python3 manage.py runserver

This should start the development server on localhost. Now we can open the project in the code editor. Also we can commit our changes to the git repository.

Your files go to Cloud : Uploading Images with Express to Cloudinary

Suyash K. — Sat, 19 Nov 2022 18:25:15 +0000

As soon as your website or app reaches a point when you realize you cannot always display static or dummy images among all the dynamic content fetched from your backend, you start looking for ways to integrate image uploads in your project. After looking around on the internet for a while, you begin to hear some names and if you aren't willing to let AWS have your card details, you will fall back to Cloudinary. Now, Cloudinary is quite convenient for small project and specially when you are still studying. But last time I tried getting it set up in my project, it wasn't the most "copy-from-docs and go" set up. Along with Cloudinary's own configuration, an important aspect is how you actually bring your files to Cloudinary's API for the upload.

In this one, we will be implementing image uploads to cloudinary wherein we will demonstrate it using a form to set profile pictures for the user. Our focus here would be on how we will handle the file so that it can be uploaded. I will use a node.js backend with authentication already implemented. In the client, I will use a form that handle user's profile information but we mostly care about the profile picture.

Preface

I will be going in depth in this one. If you need a to-the-point solution to how you can get image upload set-up with cloudinary you can skip to In short

Let's get started

So, I am trying to add a profile picture to the users in my database by uploading them to cloudinary. I would have a form in the front end, I would use a file input to select a file and then it would send this file to the backend where it would be uploaded to cloudinary. Now, these things come with hidden complexities at each steps. The file input gives us a Javascript File Object which the database or cloudinary doesn't really like or understand. We need raw image data or maybe a url. Also, JSON wouldn't send images as it is basically just fancy text. Even if we do manage to send these File Objects to our endpoint, this file object would not be enough to upload the image to cloudinary. Let's test each of these using an express server and a mongodb database.

Situation

I have a user collection in my local mongodb and I have added one user to it manually which looks like the following:

blogdb> db.users.find()
[
  {
    _id: ObjectId("629f08b134f91b645470ca97"),
    username: 'suyious',
    name: 'Suyash Kumar',
    email: 'solo@gmail.com',
    password: '$2b$10$eT7nsqWx3jVjlH9xbHOU2.EcowoHlXf8VWMldb2klFKL2CzxSeC5a',
    role: 'user',
    createdAt: ISODate("2022-06-07T08:13:37.436Z"),
    __v: 0,
    avatar: {
      public_id: 'xetra/products/vunlvjcijttmyu6aah1o',
      url: 'https://res.cloudinary.com/djv5txrzp/image/upload/v1667916852/xetra/products/vunlvjcijttmyu6aah1o.jpg'
    }
  }
]

This should give an idea of how the user collection looks like. We have an avatar field which is an object with a key url. Now I would like to change this avatar field and by changing it, I imply sending an image from my frontend to the server which would then get uploaded to cloudinary and the avatar field would have it's new url. Now we can always manually upload our images to cloudinary and just send it's url through a JSON, but we need that to happen on it's own and we should just need to use the html file input.

Sending the image from the frontend

I guess even now, you would have started getting a hint that JSON would probably not send our file object to our server. But, let's test that anyway. All we need is an <input type="file"/> and an axios.put('/upload', body). When we are sending the image from the frontend, the request body looks like the following in Javascript:

Looks fine for now but as soon as this request reaches the backend, thing begin to look different.

Clearly, JSON doesn't seem to know what a file exactly is. Guess, it's time to run to stack overflow.

Turns out, apart from application/json, there are other content-type for your request body and one of them is mutipart/form-data. In short, this is what we use when we have files along with text to send to server. Looks like that should solve our problem and the File would show up in our backend.

multipart/form-data

In order to send multipart/form-data through an HTMl form, you need <form enctype='multipart/form-data'> tag and when using axios, you need axios.put('/put', body, { "Content-Type": "multipart/form-data" }). You can also create a FormData Object in Javascript and set key/value pairs to construct your request body.

So, now we should just try sending form-data from our frontend similar to before, only this time our request body would be constructed using:

const body = new FormData(formElement);

The formElement is a reference to am html form which can be the name of a form ( or ref.current when using React.useRef)

In the frontend, the request body looks like this:

But now, we turn to our server only to see an even more obscure output.

We should also try to send the same form-data using postman just in case we are making some mistake in our frontend code.

Clearly, even postman gives us a similar output. It seems like the backend isn't able to understand what we are throwing at it. Hmm. That doesn't happen often. Or does it? One might begin to remember that we couldn't earlier even read the JSON output if we just threw it at the backend. If that didn't ring a bell, I am talking about the express.json() or earlier bodyparser() middlewares that we needed to parse our JSON requests. It is fairly obvious that we would also need to parse form-data before our backend can understand it.

Now, there are multiple similar libraries for accomplishing this and the most famous one that I can find is multer. Naturally, we would start by installing it and including it in our server. We could add the middleware for every route similar to express.json() with express.use(), but we would want to only add it for the routes that actually need form data.

multer

So, if we go forward, install multer and add a middleware multer().single('avatar') (see docs) to our upload route, we find this in our console:

Looks like our form-data did pass through. But where is the file. Turns out the files are not a part of req.body. Instead they are in req.file.

We can finally see our file reach the backend. This looks good. Now all we have to do is throw this file at Cloudinary and it should just work out of the box. Right? Well, we all know our history with throwing things around here.

Cloudinary

Now, let's install cloudinary and set it up with the configurations. You need following environment variables for the config:

cloudinary.config({
  cloud_name: process.env.CLOUDINARY_NAME,
  api_key: process.env.CLOUDINARY_API_KEY,
  api_secret: process.env.CLOUDINARY_API_SECRET
})

Above is also the code that you need to add to your server for initial configuration of cloudinary. Once this is done, you should be able to talk to your cloudinary account. ( If it wasn't obvious, you need to create an account. It won't charge you or ask for card details. You will get your api_key and api_secret from the dashboard

Now, as from the docs, we need the following to upload our image:

import { v2 as cloudinary } from 'cloudinary'
cloudinary.uploader
  .upload(file, options)
  .then(callback);

Once we are set up, we should try to use the file we received from the frontend on the upload function provided by the api. I have the following code added to my upload route:

cloudinary.uploader.upload(req.file)
  .then((result => {
    console.log(result);
  }))

This time, the server gives us a new error. Depending on how your server is set up, the error might show up at different places. As for me, the error comes as a 500 Error Response.

The error message tells us that cloudinary's API does not accept a Javascript File object and instead is asking for some string. If we look into the docs again, we find the following:

Given above, the upload function takes only the allowed types as the file parameter. Out of these, we do not have a local file path in the server as the file comes from the frontend. Neither do we have a file URL or S3 bucket URL. We are left with using a byte-array buffer or a Data URI (Base64 encoded).

As we can see above, we do seem to be getting a buffer key in the file object received. Let's inspect it's value and see if it can be of our use. If we just plug the buffer value to the upload function, the cloudinary api still cannot process it and gives a similar error.

We could try to process this byte array and make it work with the API but if we look at multer's documentation, we notice using the buffer option requires storing the images into memory which could be bad for the server in terms of performance.

Now, we are left with either trying to convert the file into a byte-array in the frontend or using the data URI option. Also, we should try to do the computations in the frontend itself as much as possible, now that we are reminded.

If we dig around a little to find some information about byte-array or Data URI, we come across the FileReader object provided by Javacript's File API. This object provides functions for dealing with both byte-arrays and Data URIs.

FileReader

This FileReader object delivers the data using events, as reading from disk may take time. The Syntax for using it is generally of the form:

const reader = new FileReader();

reader.onload = () => {
  <your-processing-function>(reader.result);
}

reader.<instance function>(blob);

We should try the readAsDataURL(file) function to send the image file as a Data URI. Note that the function is readAsDataURL and NOT readAsDataURI. The result of this function call returned through reader.result can also be used with <img src=""/> to preview the image. The following code demonstrate how we would process the file before sending it.

const reader = new FileReader();
reader.onload = () => {
  console.log(reader.result);
}
reader.readAsDataURL(input.files[0])

Here, input may be e.target if you are listening to events on a file input or inputref.current if inputref is a ref to an input element when using React.useRef. The resulting Data URI is returned as a string through reader.result. Now, that this is a string, it should work as an argument to cloudinary API's upload function.
The result looks like the following:

It is basically a bunch of gibberish to us but will represent our image file to the server. When we send this new Data URI string to the server, it gets sent as is, that is as a string. But now that it is no longer a file. It is part of req.body itself and not of the req.file

Now, we could plug this string into cloudinary API's upload function and see if it works:

cloudinary.uploader.upload(req.body.avatar)
  .then((result => {
    console.log(result);
  }))

This, after all this work, actually works and we get the following result in our server console:

We can go to the provided secure url and it is actually uploaded to cloudinary. We can also see this image in our cloudinary media library. Here, I have used the folder option that can be provided in the second argument of the upload function as follows:

const result = await cloudinary.uploader
  .upload(req.body.avatar, {
    folder: "/fraise/avatars"
  })

The API returns an object with a url for the uploaded image as secure_url, which can now be stored in the database to update our user profile.

const user = await User.findByIdAndUpdate(req.user.id, req.body, {
  new: true,
  runValidators: true,
  useFindAndModify: false
})

Here, my server is setting req.user while validating authentication, though it is only the providing the id of the entry to update in our database. This updates the current user's profile picture.

In short

In short, all I did all this while was:

send files as form-data and not JSON.

here you might need:

<form enctype='multipart/form-data'>

axios.put('/put', body, { "Content-Type": "multipart/form-data" })

receive form-data using multer.

you might need to install multer and add the middleware:

multer().single('name')

for other usages, here's the docs

setup cloudinary and upload

cloudinary.config({
  cloud_name: process.env.CLOUDINARY_NAME,
  api_key: process.env.CLOUDINARY_API_KEY,
  api_secret: process.env.CLOUDINARY_API_SECRET
})

import { v2 as cloudinary } from 'cloudinary'
cloudinary.uploader
  .upload(path, options)
  .then(callback);

send files as dataURI for the path in upload() above:

const reader = new FileReader();
reader.onload = () => {
  console.log(reader.result);
}
reader.readAsDataURL(input.files[0])

But this makes me think

Even though we used multer to be able to read multipart/form-data in the server, we don't quite seem to have used it for actual file transfer between the frontend and the backend. It makes me wonder if using form-data was for nothing. I did try to use JSON for the same but it didn't quite seem to work. Now, I didn't want to pull apart all that code again. So, I urge you to try it.

Additionally, the data URI seems to be a very long string which might be too long to send back and forth. Tell me if there are other simpler ways to upload the images. I do know of another method, creating a temporary file on the server and using it's file path for the upload. This would require creating and removing files on the server.

I would be really happy if I am pointed to a better direction here, but if my solution was helpful to you, that would make me really happy too.

Storing & Rendering Blogs: no markdown-parser

Suyash K. — Sat, 05 Nov 2022 16:14:29 +0000

I've been trying to put together a blog for myself and a point comes when one needs to start thinking about strategies to store the blogs somewhere. One common strategy you find in a lot of tutorials is using some third-party library that parses markdown to the desired semantic, for eg. react-markdown . Now, speaking of the goods of such a library, it (hopefully) tackles the security concerns that come with letting external HTML render on a site. This is the security concern that leads to dangerous function names such as dangerouslySetInnerHTML, quoting the react docs , just " to remind yourself that it’s dangerous " . On a serious note, XSS attacks can be a major pain in the posterior. Apart from that, such libraries also eliminate the need to implement your own markdown parser.

That being said, I have this little itch to NOT use such a library for my blog.

Using such a third-party library would mean adding to your dependency tree and also succumbing to making your whole site rely on its maintenance. Also, using it might be very practical for a client project but I am looking forward to learning something here and I think I am pretty skilled already in npm i <package> and copying boilerplate from the documentation.

In this article, however, we should just focus on the strategies for implementing text processing in the blog to have basic formatting capabilities and for creating a good enough schema for storing it in a database. If I succeed, you should see this become a 10-part series or more. 🤓

Preface

Before moving any further, let me clarify that although the text-parsing mainly happens in the frontend, I would also be speaking about how various things are done in a node-express backend. But this isn't meant for people who are completely new to express or backend in general. I would expect awareness about REST APIs and how it is implemented in express from the readers. Additionally, we will probably also be talking about regular expressions ( Perl-compatible ) in javascript. If you think you don't fit in here, let me point you to some resources to get you ready!

REST APIs and Express: docs, short video
Regular Expressions: docs, short video

Let's begin

We first need to plan how we store the blogs in a database. Rather than storing completely as a secondary markup language that needs parsing, I would like to store it in a format the database is more familiar with. Assuming that we are using a NoSQL document database like MongoDB, we can start drawing a rough picture of our model as a JSON. The same can also easily be translated into a relational table. However, for simplicity, we stick to JSON.

blogModel = {
  author : user
  created at: date
  title: string
  tags: [ strings ]
  image: string
  body: [
    { type:  "head" | "para" | "quote" | "image" | "code" ,
      value: string   }]
}

This roughly represents the schema for a single blog entry in our database. To summarize it: in short, we store the author, creation date, title of the blog entry, an image for the cover, and the body of the article. The focus here is the body of the entry which will be an array of individual sections of the entry. If we are talking about relational representation, this could be thought of as a foreign key to a sections table. This way we will have more flexibility while styling the elements in our frontend as they would already be isolated as headings, images, code blocks, etc. Additionally, we will add styles for text formatting and links using regular expressions.

Now that we spent our time in the back, let's think about how this would look on the front. Once the blog is uploaded, it should look typically the same as the one you are reading right now. The part we care about is the editor which will be used for sending post requests to our backend. This is the part where we need to process user's text into a form that our backend will understand. For now, this lays down a simple layout for our create-blog page.

The text processing needed for turning whatever user enters into our editor should be implemented in the client with some regular expressions magic. This should be the highlight of this project and while other things are important, this is the part that will decide whether the whole idea works or breaks.

There may be some other smaller concerns like where we plan to store the images for our blog. Here, one option would be to store it right in the database or maybe in the server and the other would be to use a storage service available, for instance, as you might have noticed, sites like dev.to seem to be using either AWS s3 or cloudinary for storing images. The latter should be a better option but it largely depends on the scale of our application. A personal blog with very few images should not require a pricey service.

With that being said, I kept this one less populated with code and more with the general plan. You can tell me if you like the idea or point out if I am missing some key aspects that might start to cause some trouble once I am deep into the project. Or, you can share any of your ideas of how a better route can be taken for what I'm trying to achieve.

Hopefully, in the next part, we will get deeper into the implementation. We will make use of regular expressions, express, and CSS/Javascript for styling.