Forem: Frank Ezenwanne

HOW TO UPLOAD A CSV FILE TO DJANGO REST

Frank Ezenwanne — Tue, 05 Nov 2024 10:05:14 +0000

Uploading a CSV file to Django REST (especially in an atomic setting) is a simple task, but kept me puzzled until I found out some tricks I would be sharing with you.
In this article, I will be using postman (in place of a frontend) and will also share what you need to set on postman for request sending via pictures.

What we desire

Upload CSV via Django Rest to the DB
Make the operation atomic i.e any error in any row from the csv should cause complete rollback of the entire operation, so we can avoid the stress of cutting the csv file i.e the headache of identifying the portion of the rows that made it to the DB and those that didn’t due to any error midway!! (partial entry). So we want an all-or-none thing !!

Method

Assuming, you already have Django and Django REST installed, the first step would be to install pandas, a python library for data manipulation.

pip install pandas

Next in postman: In the body tab, select form-data and add a key (any arbitrary name). In that same cell, hover on the rightmost of the cell and use the dropdown to change option from text to file. Postman will automatically set Content-Type to multipart/form-data in Headers the moment you do this.

For the value cell, click the 'Select Files' button and upload the CSV. Check the screenshot below

Under headers, set Content-Disposition and the value to form-data; name="file"; filename="your_file_name.csv". Replace your_file_name.csv with your actual file name. Check the screenshot below (NB:You might have to remove quotes from name and filename values in postman if you having issues with .csv parsing in the python code)

In the Django views, the code is as follows:

from rest_framework import status
from rest_framework.views import APIView
from rest_framework.parsers import FileUploadParser
from rest_framework.response import Response
from .models import BiodataModel
from django.db import transaction
import pandas as pd

class UploadCSVFile(APIView):
    parser_classes = [FileUploadParser]

    def post(self,request): 
        csv_file = request.FILES.get('file')
        if not csv_file:
            return Response({"error": "No file provided"}, status=status.HTTP_400_BAD_REQUEST)

        # Validate file type
        if not csv_file.name.endswith('.csv'):
            return Response({"error": "File is not CSV type"}, status=status.HTTP_400_BAD_REQUEST)

        df = pd.read_csv(csv_file, delimiter=',',skiprows=3,dtype=str).iloc[:-1]
        df = df.where(pd.notnull(df), None)

        bulk_data=[]
        for index, row in df.iterrows():
            try:
              row_instance= BiodataModel(
                      name=row.get('name'),
                      age=row.get('age'),
                      address =row.get('address'))
              row_instance.full_clean()
              bulk_data.append(row_instance)
            except Exception as e:
                return Response({"error": f'Error at row {index + 2} -> {e}'}, status=status.HTTP_400_BAD_REQUEST)

        try:
            with transaction.atomic():
                BiodataModel.objects.bulk_create(bulk_data)
        except Exception as e:
            return Response({"error": f'Bulk create error--{e}'}, status=status.HTTP_400_BAD_REQUEST)
        return Response({"msg":"CSV file processed successfully"}, status=status.HTTP_201_CREATED)

Explaining the code above:
The code begins with importing necessary packages, defining a class based view and setting a parser class (FileUploadParser). The first part of the post method in the class attempts to get the file from request.FILES and check its availability.
Then a minor validation checks that it is a CSV by checking the extension.
The next part loads it into a pandas dataframe (very much like a spreadsheet):
df = pd.read_csv(csv_file, delimiter=',',skiprows=3,dtype=str).iloc[:-1]
I will explain some of the arguments passed to the loading function:

skiprows
In reading the loaded csv file, it should be noted that the csv in this case is passed over a network, so some metadata like stuff is added to the beginning and end of the file. These things can be annoying and are not in comma separated value (csv) form so can actually raise errors in parsing. This explains why I used skiprows=3, to skip the first 3 rows containing metadata and header and land directly on the body of the csv. If you remove skiprowsor use a lesser number, perhaps you might get an error like: Error tokenizing data. C error or you might notice the data starting from the header.

dtype=str
Pandas likes to prove smart in trying to guess the datatype of certain columns. I wanted all values as string, so I used dtype=str

delimiter
Specifies how the cells are separated. Default is usually comma.

iloc[:-1]
I had to use iloc to slice the dataframe, removing the metadata at the end of the df.

Then, the next line df = df.where(pd.notnull(df), None) converts all NaNvalues to None. NaNis a stand-in value that pandas uses to rep None.

The next block is a bit tricky. We loop over every row in the dataframe, instantiate the row data with the BiodataModel, perform model-level validation (not serializer-level) with full_clean() method because bulk create bypasses Django validation. We then add our create operations to a list called bulk_data. Yeah , add not run yet ! Remember, we are trying to do an atomic operation (at batch level ) so we want all or None. Saving rows individually won’t give us all or none behaviour.

Then for the last significant part. Within a transaction.atomic() block (which provides all or none behaviour), we run BiodataModel.objects.bulk_create(bulk_data) to save all rows at once.

One more thing. Notice the index variable and the except block in the for loop. In the except block error message, I added 2 to the indexvariable derived from df.iterrows() because the value did not match exactly the row it was on, when looked at in an excel file. The except block catches any error and constructs an error message having the exact row number when opened in excel, so that the uploader can easily locate the line in the excel file!

Thanks for reading!!!

VERSIONS OF TOOLS USED

python==3.11.9
Django==4.2.10
pandas==2.2.3
djangorestframework==3.14.0


Also postman app for windows --> version 9.18.3

CONTROLLED VS UNCONTROLLED INPUTS IN REACT JS

Frank Ezenwanne — Fri, 12 Jan 2024 15:53:51 +0000

One thing about React is that the view is rendered based on state. Changing the state triggers a re-render. The state is essentially data that is dynamic based on user interactions. This topic in React is very fascinating as it can be likened to real life scenarios especially in some marriages.

CONTROLLED INPUTS
Before the invention of React, some HTML input elements were already used to collecting and holding dynamic data by themselves. These elements are like women that are already making their money before marriage. When they get married to their husbands i.e when the elements get into the realm of React as Jsx, the husband i.e React tries to convince them to go with a single source of truth ....Sounds weird right?

This means every money they make is transferred up to their husband's account . If they need money for food, taking care of the home etc, the husband transfers money down to them. Every data the input elements collect is not exactly collected by them, they pass it up to the React state which then passes it down to them.

EXAMPLE

import React, {useState} from "react";

const ControlledInputComponent= (props)=>{
    const [username, setUsername] = useState('')

    const handleChange= (e) => {
       setUsername(e.target.value)
    }


  return (
     <input type type='text' value = {username} onChange = 
     {handleChange} />
   )

}

The example above shows that the value attribute of the input which is supposed to be the direct input from the user, is actually a let down input from the state. The onChange attribute is in charge of taking every data up. So sadly, it all goes up.
Though, there might be no advantage in the real life scenario, except for men who want super-submissive wives, but in the React sense there is an advantage and of course a disadvantage.

Advantage

This approach is cool when you want features like automatic checks and transformation of the UI as the user progressively types. This is because the state is updated on every entry allowing for a new UI to be rendered.

Disadvantage

This approach although used a lot, might be draining and take a toll resulting in performance issues. Naa not sexual performance, but browser performance due to frequent re-rendering of heavy UIs

UNCONTROLLED INPUTS
The input elements here, hold on to their values.The women don't give it all up to their husbands..The elements don't pass input data up to React state. The browser, like their personal bank account, holds on to the data even after marriage to React. The elements do this with the help of a ref. Refs can be gotten from the useRef hook in React.

EXAMPLE

import { useRef } from "react";

function UncontrolledInputComponent() {
  const newRef= useRef('');

  const handleSubmit = (e) => {
    e.preventDefault(); //prevents page from refreshing
    console.log(inputRef.current.value); //here we log out the value from the current attribute of the ref object
  };

  return (
    <form onSubmit={handleSubmit}>

      <input type="text" ref={newRef} />

      <button type="submit">Go</button>
    </form>
  );
}

REDUX OR ZUSTAND?

Frank Ezenwanne — Sun, 02 Apr 2023 22:22:56 +0000

𝐑𝐄𝐃𝐔𝐗 𝐎𝐑 𝐙𝐔𝐒𝐓𝐀𝐍𝐃 ?
As a React developer, I have been trying to compare Redux with Zustand . I started my React dev journey with Redux, but along the line, I met Zustand! At first, I thought it would be weird taking Zustand on, but it turned out to be great!

Redux has quite a lot of boilerplate code and requires setting up different 'departments', in terms of actions, reducers, combine-reducers etc ....giving some sort of decentralized vibe. The redux logic flow also involves 'funny' patterns like importing an action, not using it yet, passing it into the connect binder which returns a 'prop action' possessing affiliation with the store, that your component actually uses!

Zustand on the other hand, puts all these 'departments' into one file and uses react hooks, hence is preferably used with function based components.

At first, I still wasn't impressed after having a look at the simplicity of Zustand. I still thought about the react-redux connect binder which lets React components see the redux props as 'part of its state' (prolly in a secondary manner) hence inducing a re-render...hot reload! Well, recently I think I've seen a way Zustand can support that !

So yeah, though Redux might have some in-built advanced features like time-travel debugging, Zustand might just be all you need! Still have a thing for Redux though.

WHY YOU NEED TO SET TOKENS IN WEB APPS FOR AUTHENTICATION IN DJANGO REST AND NOT DJANGO ( 𝘚𝘐𝘔𝘗𝘓𝘐𝘍𝘐𝘌𝘋 )

Frank Ezenwanne — Thu, 06 Oct 2022 01:14:00 +0000

𝐍𝐎𝐑𝐌𝐀𝐋 𝐃𝐉𝐀𝐍𝐆𝐎 :
All the frontend pages are rendered by the backend, so since the backend knows that the frontend pages are rendered by it, it can easily pass a user object into the template page. The template language can allow you to test for logged in status with 'if user.is_authenticated' right inside the frontend( the html). So it’s like a piece of the backend is sent to the frontend in a format that can be accessed by a special template language e,g Jinja2, not a general format like JSON(take note) as it renders on the user’s browser.

The fact that a piece of the backend is sent with the frontend already puts in a level of security as it would be a bit difficult to tap into the data and change 'is_authenticated' to True. It’s like the frontend is ‘pampered’ here!

𝐃𝐉𝐀𝐍𝐆𝐎 𝐑𝐄𝐒𝐓:
Naa…No pampering here! The frontend is decoupled from the backend! By decoupled, I just mean the rendering work is handled by the frontend as in the case of Javascript frameworks like React which have the concept of 'state'(I'll talk about it someday). The frontend pages are manufactured/handled by React. The Backend just brings the raw data.

The frontend says, ‘Yeah just drop it there, I’ll do the rendering myself. I’m old enough now!’ This attitude is why the backend cannot just trust the frontend by dropping the same 'is_authenticated' attribute. Nope! Instead, it gives a token, which is a very long alphanumeric set of characters that cannot easily be forged. This token will allow the frontend to access data in the backend. The backend can identify the user with the token, even the expiry date info is embedded. You might say, the attitude of the frontend does not deserve this strict behaviour by the backend, but remember, that the pages are no longer brought by the backend.

The 'is_authenticated' attribute if it were used, would come as JSON format as opposed to the special format in normal Django. JSON is like a format many programming languages see as a meeting point. So imagine putting in something as simple as {is_authenticated : true} . From the frontend, using a simple API tool, one can easily tap in and send as JSON, {is_authenticated : true} as an attribute back to the backend and the backend will open its doors wide if that is the pass.

How To Upload Image and Text from React to Django with JSON and Proper Encoding.

Frank Ezenwanne — Wed, 25 May 2022 14:34:12 +0000

Uploading only text as JSON from React to Django is quite straightforward. Django serializers easily convert JSON to python native data types. JSON is just like python dictionary in strings(quotes).

But then, how can we send images along with text?. Well, sending image files fixed in the react component state by e.target.files[0] did not work.

A little research brought up the idea of encoding the images as base64!!😈

Okay..okay, Don't let that scare you.

Two or three lines on the frontend, we are done. We don't even need to install anything. Same thing on the backend, 2 lines and we just need to pip install a little package.

We don't even need any header on the frontend, except you are sending an authorization token. We don't even need parsers in the API.

Highlights.

NB: This is not a tutorial on setting up Django with React. A good tutorial on that would be a YouTube series by Brad Traversy (Traversy Media), Full stack React and Django.
For pure Django, then Django Tutorials by Corey Schafer.

We are gonna be using a Post model as an example (Like a Blog Post).

THE BACKEND (DJANGO REST)

Create the Post Model
Adding MEDIA_URL and MEDIA_ROOT to settings.
Pip installing drf_extra_fields (the only installation)
Create the serializers
Create the API
Set up the url.

THE FRONTEND ( REACT )

Set up the PostCreate component with state and onChange attribute.
Adding the Image change handler, conversion of the image to base64 with FileReader.
Setting up axios and sending the JSON data with the base64 representation.

NOW LET'S GO INTO DETAILS

THE BACKEND

1. Create the Post model

We will start by creating the Post model

from django.db import models
from django_resized import ResizedImageField
from django.utils.text import slugify
from django.utils import timezone
from django.urls import reverse
from django.contrib.auth.models import User


class Post(models.Model):
    title = models.CharField(max_length=150)
    slug = models.SlugField(blank=True)
    file = models.ImageField(null=True,upload_to = 'post_pics',blank=True)
    date_posted = models.DateTimeField(default = timezone.now)
    content = models.TextField()

    def __str__(self):
        return f'Post : {self.title}'

    def save(self, force_insert=True ,*args , **kwargs):
         if not self.slug:
            super().save(*args,**kwargs)
            pk=str(self.pk)
            slug_field = slugify(self.title) + pk
            self.slug = slug_field
            return super().save(*args,**kwargs)
         return super().save(*args,**kwargs)

    def get_absolute_url(self):
        return reverse('post-detail',kwargs ={"slug":self.slug})

The image field takes null=True to allow image upload to be optional.

2.) Adding MEDIA_URL and MEDIA_ROOT to settings.

Next, we'll add MEDIA_ROOT AND MEDIA_URL to Django settings.py to enable us to create a local storage location for the uploaded images.

MEDIA_ROOT = os.path.join(BASE_DIR,'media')
MEDIA_URL = '/media/'

3.) Pip installing drf_extra_fields (the only installation)

This is the only installation we will be doing in this tutorial. We need to use the Base64ImageFieldin the package to accept the base64 data.

pip install drf_extra_fields

4.) Create the Serializer class

from rest_framework import serializers
from .models import Post, Review

from drf_extra_fields.fields import Base64ImageField

class PostSerializer(serializers.ModelSerializer):
    file = Base64ImageField()

    class Meta:
        model=Post
        fields= ('title','file','content')

Notice how the file field was set to be the Base64ImageField. The field will receive the base64 data and will allow for conversion back to an image.

5.) Create the API

from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework.permissions import IsAuthenticated
from .models import Post
from .serializers import PostSerializer, 


class PostAPI(APIView):
    permission_classes = [IsAuthenticated]
    def post(self,request,*args,**kwargs):
        serializer = PostSerializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        instance = serializer.save()
        response = {        "title":instance.title,"content":instance.content,"date":instance.date_posted.strftime("%a %H:%M %d/%m/%y"),"file":instance.file.url,
            "url":instance.get_absolute_url()
        }
        return Response(response)

The api gets the JSON data from the frontend, passes into the serializer, which validates and saves the data with the base64 being converted back to an image. Finally it accesses the properties of the saved instance including the url of the saved image and sends it back. I am not sending back the image to the frontend, but rather, a link to the saved image on my local drive.

You might be thinking why not send back as base 64.. Well, that would mean I can't open up the storage location and view. Also, in the frontend, I'll have to convert again from base64. So I didn't bother. So I think it is better this way.

6.) Set up the url.

from django.urls import path
from .api import PostAPI


urlpatterns=[
    path('api/create',PostAPI.as_view()),]

Here, we set up the URL necessary to link the react request to the api.

That's it for the Backend..

THE FRONTEND (REACT)

1. Setting up the PostCreate component with state and onChange attribute.

import React, {Component} from "react"
import axios from "axios"

class PostCreate extends Component{

    state = {
        title: "",
        content: "",
        file: ""
    }


    onchange=(e) =>{
        this.setState({[e.target.name] : e.target.value})
    }

    render(){

        const { title, content, file} = this.state

        return(
            <div className = 'create-form'>
                <h4 align="center" className = 'createpost-heading'>Start A New Topic</h4>
                <span className ="create-post-requirement">A Title is Enough To Start.. </span>
                <form onSubmit={this.onsubmit}>
                    <div className = 'form-field'>
                        <span className= "asterik-field">*</span>
                        <label className= "post-create-label-first" htmlFor = "id_title">Title</label><br/>
                        <input id = "id_title"
                        className = 'user-field'
                        type ='text'
                        name ='title'
                        placeholder=' Title of Post'
                        size = '110'
                        maxLength = '100'
                        value = {title} 
                        onChange={this.onchange}
                        />

                    </div><br/>


                    <div id="post-create-text" className = 'form-field'>
                        <label className= "post-create-label" htmlFor = "id_content">Write Something</label>
                        <textarea id = 'id_content'
                        className = 'content-write'
                        type ='text'
                        name ='content'
                        placeholder=' Write post content'
                        rows = '7'
                        cols = '25'
                        value = {content}
                        onChange={this.onchange}>
                        </textarea>
                    </div> <br/>

                    <div id="post-create-image" className = 'form-field'>
                        <label className= "post-create-label" htmlFor = "id_postimage">Upload A Game Pic</label>
                        <input id = "id_postimage" 
                        className = 'post-image-field' 
                        type ='file' 
                        accept = 'image/*'
                        name = 'file' 
                        />
                    </div><br/>

                    <button type = "submit" className = 'form-button'>Submit</button><br/>

                </form>
        </div>

        )
    }
}


export default PostCreate

Here, we have created the component for post creation and put in the fields. We have also set the onChange handler for the title and content fields.

2.) Adding the Image change handler and conversion of the image to base64 with FileReader.

Now let's set up the handler for the image field. You'll see the base64 encoding by FileReader in action here.😈

imageChange = (e) =>{
        const file = e.target.files[0]
        const reader = new FileReader()
        reader.onload = () => {
            this.setState({file : reader.result})
        }

        if(file){
            reader.readAsDataURL(file)
        }

    }

Now, what happens here is very simple. The first line gets the uploaded file under the file variable. The next line creates a FileReader object. Let's visit the last block before the reader.onload. The last block calls reader.readAsDataURL on the file and converts it to base64 format. The reader.onload runs an arrow function when reader.readAsDataURL is triggered to handle a file i.e just like an event Listener. The arrow function simply sets the state with the base64 file.

3.)Setting up axios and sending the JSON data with the base64 representation.

We are gonna be setting up the axios in the onSubmithandler function, so that the axios request is triggered on submission.

onsubmit = (e) =>{
        e.preventDefault();
        const {title,content,file} = this.state
        const token = localStorage.token
        let config={}

        if(token){

            config = {
                "headers": {"Authorization":`Token ${token}`
                  } 
            }
        }

        const body = {title,content,file}
        console.log(body)
        axios
        .post("api/create",body,config)

        .then(
            (res) => {
                console.log(res)
            }
        )
       .catch(
            (err)=>{
                console.log(err.response)
           }
       )
    }

Token was used for authentication explaining the token setting in the header. Using JSON.stringify wasn't necessary on the body before sending. The operation is quite simple. After preventing default submission with e.preventDefault , the fields were extracted from the state, and token, from localStorage. The axios request comes in to finish the job by sending the data and handling success with .then, and failure with .catch

VERSIONS OF TOOLS

 Python == 3.7.2,
 Django == 3.2.13,
 drf-extra-fields == 3.4.0,
"react": "^18.0.0",
"react-dom": "^18.0.0",
"webpack": "^5.70.0",
"webpack-cli": "^4.9.2",
"axios": "^0.27.1",
"react-router-dom": "^6.3.0",
"@babel/core": "^7.17.9",
"@babel/preset-env": "^7.16.11",
"@babel/preset-react": "^7.16.7",
"babel-loader": "^8.2.4",
"babel-plugin-transform-class-properties": "^6.24.1"

And that's a wrap! I hope you enjoyed the article. I'd love to read/hear your comments. 😊