Forem: Farhan Masud Aneek

Setting up Django in a Better Way and Understanding How It Works

Farhan Masud Aneek — Mon, 13 Nov 2023 18:36:28 +0000

When you are setting up a new Django project, you'll have to configure quite a lot of things depending on the requirements of the project. Starting from database configurations to a custom user model, there are more than quite a few things to be done in between.

There are very useful packages for bootstrapping your Django projects in minutes such as django-cookiecutter and djangox. If you are a seasoned developer, I'd highly recommend using one of these instead of what I'm going to show here. But if you are struggling with the project structure of these packages as a beginner to intermediate Django developer and looking to structure your own Django projects in a better way, I have created a lightweight setup that deals with the basics of setting up a Django project with PostgreSQL as database and TailwindCSS as our styling library.

This Django Starter kit takes care of automated creation of virtual environment and installing of Python packages and setting up the database with bash scripts. In addition to PostgreSQL and TailwindCSS, all the sensitive values are taken care of in a .env file using django-environ package. The virtual environment is maintained using pip-tools.

The static files are managed with whitenoise, Gunicorn is used for the WSGI server and Sentry is used for error monitoring and reporting in production.

Since the scripts are in bash, these will only run on UNIX based operating systems like Linux and MacOS. But you can also run this on Windows with Git Bash or WSL.

Before getting started

Make sure you have PostgreSQL and Node.js + npm installed on your local machine.

Quick Setup

Here are the steps to setup this starter kit. Setting this up shouldn't take more than 5 minutes. I'm going to explain all the steps in detail and how it's configured on the later sections so that you have in depth understanding about how this works under the hood (which will take way longer than 5 minutes but I can assure that it's worth the time). By understanding the structure of the project from the later sections, you can create something on your own that suits your needs best.

Clone repo with your project name git clone git@github.com:farhanmasud/django-tailwind-starter-template.git your-project-name
Setup .env file following the example env file example.env
Make all bash scripts executable with find . -type f -iname "*.sh" -exec chmod +x {} \;
Run bash update-git-remote.sh script to remove existing git remote link (of this repo) and update with your on git repo link. Copy your remote URL from GitHub and run the bash script, it'll prompt to enter the new link. Paste your new link and hit Enter.
Setup database [requires step 2] with bash setup-db.sh
Setup venv and install dependencies with bash setup-venv-pip-tools.sh
Activate virtual environment with source venv/bin/activate
Install tailwind dependencies with python manage.py tailwind install
Run migrations with python manage.py migrate
Collect static python manage.py collectstatic --no-input

Setup Explanation

1. Cloning the repository

First, clone this repository on your local with git clone git@github.com:farhanmasud/django-tailwind-starter-template.git your-project-name command. If you are going to call this project quick-django-project, the command will be -

git clone git@github.com:farhanmasud/django-tailwind-starter-template.git quick-django-project

2. Updating the `.env` file

Navigate to this newly created quick-django-project directory and copy the .example.env and paste with the name .env in the same directory.

Open the .env file and update the environment variables as follows.

WORK_ENV=local Since we have different settings file for different environment such as production, testing, staging and local development. We're updating this to local to make sure only packages needed for local development packages are installed and only settings that are intended for the local development environment is loaded.
Generate a secret key for Django and update the SECRET_KEY variable. You can generate Django secret key by following this question from StackOverflow or use this web UI. Let's say our generated secret key is @6jc16lc(+mhjnkfk@^)46xc728u0j&(z)y!qkimo4b#ggnos7 and in our environment file, it'll be SECRET_KEY=@6jc16lc(+mhjnkfk@^)46xc728u0j&(z)y!qkimo4b#ggnos7
Set ALLOWED_HOSTS=localhost,127.0.0.1 which will allow us to access our Django project on local machine with localhost or 127.0.0.1.
Grab the password for the user postgres on PostgreSQL and use with PGPASSWORD variable. If your password is examplepostgrespassword1234 set PGPASSWORD=examplepostgrespassword1234.
Set your database name, database password and database user as your preference.

DB_NAME=django_project
DB_USER=django_user
DB_PASSWORD=exampledatabasepassword1234

And keep the following as is -

DB_HOST=localhost
DB_PORT=5432

Create an account on Sentry and create a project for Django. You can follow the steps on Sentry documentation for Django to get your dsn link and update the SENTRY_DSN variable. Enter the value without quotes here.
SENTRY_DSN=https://examplePublicKey@o0.ingest.sentry.io/0

Setting up the .env file is done. Now let's use these environment variables to setup our Django project.

3. Making the helper bash files executable and running the bash files

There are 4 bash scripts in this starter kit -

setup-db.sh script is for setting up the database for our Django project with PostgreSQL
setup-venv-pip-tools.sh script is for setting up the virtual environment using pip-tools and installing the packages.
pip-tools require a few commands to compile and then install a new package. pip-install.sh script runs these commands one by one.
For updating the git origin, you can use update-git-remote.sh.

To make these bash script, open your terminal, navigate to the project directory and run find . -type f -iname "*.sh" -exec chmod +x {} \; Run bash update-git-remote.sh

Now the bash scripts are ready to run.

To start, create a new repository on GitHub and create an empty repository. Copy the git remote origin link and on your terminal, inside your project directory, run bash update-git-remote.sh. This will prompt you to enter your git remote origin link, just paste it here with CTRL + Shift + V or CMD + Shift + V and press Enter. Your project is now linked with your GitHub repository.

4. Setting up the Database

We're going to use PostgreSQL for the database here. We'll need to setup the PostgreSQL database and user and tell Django the database, user and user password so that Django can connect to the database. For both of the purposes we'll just update the .env file with the variables (we've already done this on step 2) and run the setup-db.sh bash script.

Similar to running other bash scripts, open up the terminal, navigate to your project directory and run bash setup-db.sh which will ask for your sudo password of your current user. Press enter after entering the password and the script will setup the database.

5. Setting up the Virtual Environment

This repository has all the packages that you'll need to get up and running with PostgreSQL, Tailwind, managing environment variables, managing static files and so on. To install this package in a virtual environment, this package uses pip-tools. pip-tools requires a couple of commands to collect, compile and install the packages that you need. We have a bash script to create our virtual environment and run all of the commands one by one.

Open your terminal and navigate to your project directory and run bash setup-venv-pip-tools.sh. This will prompt you to enter if you are in development environment or in production. Enter D if you are in development environment or enter P if you are in production environment and press Enter on your keyboard. This will create a virtual environment called venv, install pip-tools and install all the packages required for your development or production environment.

6. Adding further packages

The existing packages in this repository are for getting the Django project up and running only. You'll definitely need a lot more packages while developing the project. The packages are kept within the .in files inside the requirements directory. You'll find three files in the directory -

requirements.in for packages that will be used both in local and production such as django itself.
requirements-dev.in for packages that will only be used in development such as django-debug-toolbar.
requirements-prod.in for packages that will only be used in production such as Gunicorn.

When you need to add a new package, figure out where the package will be used and and the package there.

Then run the pip-install.sh bash script using bash pip-install.sh command, enter D if you are on your local environment or enter P if you are on your production environment. This will automatically compile the packages and dependencies and install them in the virtual environment.

7. Installing Tailwind

Most of our installation is done but we haven't configured Tailwind yet. django-tailwind is already installed with the required Python packages but we'll need to compile this to fully setup Tailwind and make it ready to work in our local development environment.

To finish the Tailwind configuration, activate the virtual environment that we have created with source venv/bin/activate and run python manage.py tailwind install to complete the installation. You must have Node.js and npm installed for this to work.

You'll see a Django app in the project directory called theme and this is the app for Tailwind. Inside this you'll find the static_src>tailwind.config.js where you can configure Tailwind as per your requirements. You'll also see there is a base template (theme/templates/base.html) which is provided by the django-tailwind package. You can use this or make your own following the conventions shown here for loading tailwind. First, you'll have to load the tailwind template tags with {% load tailwind_tags %} at the top of your html file and use {% tailwind_css %} to load the CSS files into the head of your html.

8. Running Migrations

You are probably aware that Django recommends setting up a custom user model for your project before you do any migrations. This starter kit comes with a custom user model that uses email instead of the default username field. The migrations are already made in the accounts app and there you'll find the custom user model Account.

Now we just have to apply these migrations to our database including all other default migrations that Django comes with. With virtual environment activated, run the command python manage.py migrate to apply the migrations.

9. Managing Static files

We have some static files coming from the Tailwind package. In your development environment, these will be automatically available when you start the development server. But in production, you'll need to collect the static files using python manage.py collectstatic command.

10. Run the development server!

Okay, we are done setting the Django project! With virtual environment activated, run python manage.py runserver and the development server will run on the default port 8000 on your localhost. If go to localhost:8000 from your browser, you'll see a 404 Not Found error. But don't worry, this is intentional. This starter kit doesn't come with extra app or template that you'll not need later. Just create your app from here, configure urls and start building models and views as you want.

Structure of the project

Splitting the settings.py file

When you create a new Django project, you'll find a settings.py file inside your project directory that holds all the settings file related to your Django Project. But in this starter kit, you'll see that there is a directory called config and inside this there is a directory called settings. Inside this directory you'll find multiple settings files - base.py, local.py, test.py, staging.py, production.py and engage.py. The base.py settings file holds all the settings that are applicable no matter which environment your project your running. local.py, test.py, staging.py and production.py files contain all the settings related to Local, testing, staging and production environment. Finally, the engage.py file dynamically loads the settings from local.py, test.py, staging.py and production.py based on what you have provided on the WORK_ENV variable in the .env file and combines the settings with base.py.

The reason behind this splitting is that we can safely use packages and related settings only where we need. For example, this starter kit has the package django-debug-toolbar. This is only intended for your development environment and not for your production. This can be very risky if used in production because if your Django project encounters errors, all the debug info will be shown to the user which is a severe security risk. Similarly, for tracking errors in production, we're using Sentry which is not needed in our local environment since we already have django-debug-toolbar. For keeping these settings file separate so that they don't conflict with each other, the settings file is split for serving different environments.

Keeping sensitive data out of the codebase (and version control system)

Data like SECRET_KEY or your database connection details are sensitive and can cause serious security issues if they are not protected. If you keep these (along with other sensitive data like API keys) in your settings.py file and commit them in your version control system such as Git, your project can be at risk in production.

To protect these data, it's best to keep this in a separate .env file and keeping it out of the version control system. In this starter kit, we are using django-environ to achieve this. For example, you can see in config/settings/base.py, we are reading the SECRET_KEY with django-environ as SECRET_KEY = env("SECRET_KEY") where we have a variable named SECRET_KEY in our .env file. Obviously, we have to import django-environ and the python os package with

import environ
import os

and configure it with

env = environ.Env()
env.read_env(os.path.join(BASE_DIR, ".env"))

Make sure that your .env file lives at the root of the project directory and you are good to go.

Using pip-tools

When you are working on a Python project, it's best to pin your Python packages in a separate requirements.txt file with the exact version pinned and install them in a virtual environment. Python venv is probably the most popular choice due to it's very easy setup process. Keeping required packages pinned with versions helps in many ways. For example maybe you were working on a project 2 years back and you were using a package that had the version 1.0.0 at that time. Now after 2 years, that package might have been updated to version 3.0.0. If you try to install the version 3.0.0 and try to run your project, there is high chance it won't work because it has changed.

Similarly, if you have that old project that used the package version 1.0.0 and you installed the package globally and try to start a new project with the same package and use version 3.0.0, only one of them will work. If you install version 3.0.0, the one that uses version 1.0.0 won't work and vice-versa.

Python virtual environments solve this issue. You can create virtual environments with venv and keep a simple requirements.txt file with packages and version information that's required in your project. And you can install them in the virtual environment independently so that they don't affect other projects.

Instead of venv, we are using pip-tools in this starter kit. pip-tools take things further in dependency management. Check out what pip-tools does in their official GitHub repo. In short, it helps your project find the best match for the dependent packages. For example, you might need two packages A and B in your project that requires same package C under the hood. But A requires any version of C from 1.0.1 to 1.0.10 and B requires any version of C from 1.0.7 to 1.0.15. Pip tools will automatically compile the version of 'C' that suits for both of your packages.

To keep all the requirements, you'll find a requirements directory in the project. There will be 3 files (two more after they are compiled but we don't keep that in the version control) - requirements.in, requirements-dev.in and requirements-prod.in. The idea here is similar how are splitting the settings.py file. We are only keeping the packages that are relevant to the related environment. So when you setup the virtual environment with bash setup-venv-pip-tools.sh or install the packages with bash pip-install.sh, you are prompted to enter your choice between development and production environments and with that the bash script compiles the packages with pip-tools from the requirements.in file as the base and from requirements-dev.in or requirements-prod.in as per your choice.

Custom User Model

As Django recommends, we are using a custom user model that you can find as Account model in accounts/models.py. Here we are using the user's email field as the unique identifier instead of the default username. We are also using a custom manager for managing the objects. And in accounts/admin.py, the Account model is made compatible with the Django admin so that you can add users from the Django admin.

Finally, Bash Scripts

What we have done setting up the Django project with this starter kit is updating the .env file and running some bash scripts. Bash is a fun scripting tool for UNIX based systems (although it can vary sometime from different UNIX based systems). What is done in the scripts is just telling the computer to run a series of related commands one after another. Bash scripting is a lot of fun (and frustrating) as you can automate many of your workflows with this. If you are interested, you can start learning bash scripting from here.

Well, that's pretty much all. Ended up longer than I had expected even after skipping some details I initially wanted to include. But I hope it helps you give a good insight about structuring a Django project. If you examine the repository on Github, you can get a better idea of what's actually going on which is actually very simple. And from there you can explore django-cookiecutter and djangox to see what suits your needs best and might even create something that works the best for you.

Developing Maintainable Django Projects for the Long Run

Farhan Masud Aneek — Sun, 01 Aug 2021 22:21:43 +0000

Django being the web framework for the perfectionists definitely comes with many handy features out of the box. Even after being a very opinionated framework [there is always the "Django way" for doing things], in many cases there are more than a few ways to do the same thing. This might lead to some poor structure and coding choices for many beginners as described in this blog post. These mistakes [along with many others] often lead to unmanageable code in the long run when the size of the project keeps growing.

If there is a bug and causes downtime in production then fixing the error might feel like finding a needle in the haystack with a poorly structured codebase. The above mentioned link can work as a style guide [like many other, scroll to the end of this blog post for more Django style guides] for structuring Django projects which can help manage our code better. On top of this, let's dive into 11 ways that will definitely make life easier developing and maintaining a Django project for the long run.

1. Code Formatting and Linting

The codebase should be clean, concise and readable as suggested by the Django Best Practices guide. To maintain such a codebase, the best approach is to follow a specific code style and format. The hard part is remembering all the formatting rules and getting every developer working on the project onboard with these specific rules. That can eat up a lot of unnecessary time in development.

Enter black - The Uncompromising Code Formatter. It supports all major IDEs and upon saving a file, it does it's black magic to convert all the messy code to beautifully structured code. You can also run black against your whole codebase for formatting all of the files as well. Here is a handy guide on dev.to for setting up black in VSCode.

To help you with syntax errors, pylint is an amazing tool. There is also a dedicated Django extension for pylint called pylint-django. Pylint is the default linter in VSCode and you can setup pylint-django just by installing it via pip and updating the settings as shown on here.

Finally, to help you code even faster, you can use Kite for AI powered auto-completion.

2. Environment Variables and Django Settings File

Your environment variables like SECRET_KEY or anything related to database setup should never be placed directly in the Django settings file and in the version control. These are sensitive data and you should safely keep them out of anyone's reach who's not supposed to see it.

You can use django-environ for serving the purpose. The process is very straightforward as well. You just need to create an .env file and keep all your variables there and import them into your settings file using the API of django-environ package. And keep this .env file out of version by placing it in the .gitignore file. But you should keep an .env.example file in the version control that contains a template of the original .env file.

Your development coding environment will obviously differ in one way or another and to match with that, your project's settings file will differ as well. If you are keeping your development settings in the old fashioned local_settings.py file and keeping out of version control, you are probably doing it wrong and there are many valid reasons for that.

A better approach is to split up your settings file into base, development, test, stage and production files along with a __init__.py file that will go to a settings directory replacing the settings.py file. The base.py settings file will hold all the settings that don't need to change and the development, test, stage and production settings files containing all the environment specific settings with start with importing all the settings from the base.py file as from .base import *.

One issue that will come up with this approach is that the manage.py will be pointing to the older settings file and it'll no longer work. In this case you won't be able to run your development server along with any commands with manage.py. To solve this, you can create a new environment variable called WORK_ENV in your .env file and a new engage.py file in the settings directory which will contain -

from .base import *

working_environment = env.str("WORK_ENV", default="development")


if working_environment == "production":
    from .production import *
elif working_environment == "stage":
    from .stage import *
elif working_environment == "test":
    from .test import *
else:
    from .development import *

And finally, update your manage.py file to replace the line

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project.settings')

with

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "your_project.settings.engage")

So, previously all the settings with all sensitive information would have been stored in a single settings.py file. And now, with the split up settings file and the .env file, it should look like -

|--settings
   |-- __init__.py
   |-- base.py
   |-- engage.py
   |-- development.py
   |-- production.py
   |-- stage.py
   |-- test.py
   |-- .env
   |-- .env.example

3. PostgreSQL Everywhere

Django comes with a simple and lightweight SQLite3 database out of the box which takes you out of the hassle of setting up the database and you can go straight to developing your application. But there are many cases where it might go wrong and even restrict you from writing efficient code for filtering your data from the database. Among others, one example can be how to filter distinct values as explained here, you can't really use the SQLite3 database for this.

PostgreSQL database works fantastic with Django. It is recommended by many to use PostgreSQL in production. A better approach would be using it in all of your environments [development, test, stage and production] so that your application works consistently everywhere. If you can just go through the setting up the PostgreSQL database on your environments [just a few commands anyway and if you use Docker you can streamline the process even further], you don't have to spend time thinking about which Django ORM features that you can use.

4. API Documentation

If you are working with Django REST framework, it's a good idea to keep your API documented even if you are not exposing it to the public. One great tool is Swagger and you can generate the Swagger schema automatically for your DRF project by using drf-yasg. Very easy to setup and the schema will be generated from your urls.py files. You can also choose between two different looks.

Keep in mind that drf-yasg generates Swagger/OpenAPI 2.0 specification of your REST API. If you are looking for Swagger/OpenAPI 3.0, you can check out drf-spectacular.

5. Django Debug Toolbar

The django-debug-toolbar package makes debugging a breeze. Django's default debug mode is definitely a very useful tool but django-debug-toolbar is a step above. You can see how much CPU time it takes to process the response, how many [and what] SQL queries are being executed, which static files the request is fetching, details about your template, list of all of your context variables and many more.

6. Django Shell and Management Commands

Django shell is like a Python shell but with everything that Django is setup with on your project settings. You can call all of your apps, models and anything that's defined in your project really. Just type in python manage.py shell in your virtual environment and you're inside the Django shell. This can come in handy for testing different ORM queries with your defined app models as well as trying out APIs of third party packages.

You can use custom management commands with manage.py for doing tasks. For example, you might need to insert data into your database from a csv file or a daily cron job for doing something that is related to the database. You can't access the database with the Django ORM from outside the application [for example, a cron job]. But if you define your task in a management command, you can call it from anywhere with the Python from that project virtual environment using python manage.py your_command.

There are many useful management commands can already be found in different third party packages. One such package is django-extensions. It comes with many helpful commands that can help you in many different scenario in development. You can drop the whole database, get a list of all of your urls and many more. One feature I particularly like is that I can generate a visual representation of my models using this command. When the codebase grows big and I need to get an overview of how all of my models are connected, it comes in very handy.

7. Logging with Sentry

Well, the night is dark and full of (t)errors. And you can't just play Jon Snow when the users start complaining about getting server errors.

In development, when your debug mode is on, you can just go through the very helpful debug messages that Django helps you with. But it's not the same in production. You should definitely setup logs in your Django projects but additionally, a third party tool like Sentry can save you a ton of time and hassle. The setup process can't be simpler, you just sign up and add a few lines to the Django settings file and that's it!

To get you started, the free tier gives you to save 5K errors, 10K transactions and 1GB attachments for 1 user with 30 day data retention. And you don't need to wait for your users to start complaining that your site isn't working, Sentry will send you emails when it catches new errors.

8. Environment Management - pip-tools and Docker

You should always pin the exact version of Django and other pip packages that you are using in your Django project in a requirements.txt file. You can easily do it with virtualenv and a pip-freeze command. But the issue that comes up with this is that it exports all the package names that are in your virtual environment. There can be many dependencies of a specific package and if you want to upgrade one, you'll need to update all of it's dependencies manually as well along with it. Also, there are packages that you need only in development like black, pylint or django-debug-toolbar that you definitely don't need in production. To deal with all these, there is an awesome package called pip-tools.

With the help of this package, you only need to pin your packages in a requirements.in file and when you compile this file, it'll generate a nicely formatted requirements.txt file that will have all the dependencies with notes for which one of them is coming from which specific package. You can also pin your development packages in dev-requirements.in file and keep your development tools separate.

Now you have taken care of the version of the pip packages, how about your Python version? Or exact PostgreSQL version that you are using in your development? Or even further, the version of your operating system that's all of these are running on?

When you run your application, it's behavior depends on everything that it's using [and not using, that are running alongside it on the same operating system]. All of these can influence how it's running and performing. To isolate the application and to make sure it works the same no matter where it's running, you should use Docker.

Getting started with Docker can easily feel overwhelming for a newcomer. Here is an awesome article how you can user Docker with your Django project. This is a three part series and it walks you through how you can use Docker in your development as well as deploying it live.

9. Using UserAdmin in the Django Admin for Custom User Model

One of the most handy "batteries" included with Django has to be the Django Admin. It can be very useful for inspecting the data in the database with a model-centric interface as well as for inputting / inspecting / managing test data while in development. When we add a model to the Django Admin, the usual approach is to use ModelAdmin from django.contrib.admin but if you use it with a custom user model, there is an issue. And if you are not already aware, defining the custom user model should be something that you should do as one of the first things when you start a new Django project as suggested by the Django documentation itself.

Just like most other web frameworks, Django doesn't store the passwords as plain texts. It uses a hash function to encrypt it and then stores it in the database. So even if the data is compromised, it won't be possible to decipher what actually the password is for a specific user.

Going back to the issue about using ModelAdmin with a custom user model is that your user create form and user edit form will present you with password input fields where you'll need to enter the password in the hashed format itself. If you enter yourawesomepassword123 as your password and save the user, that user won't be able to log in with yourawesomepassword123 because that's not the actual stored password. The actual password would be the value that's is yourawesomepassword123 after hashing.

The solution is to write user creating and updating forms based on UserCreationForm and UserChangeForm from django.contrib.auth.forms and use them in the admin that will be based on UserAdmin from django.contrib.auth.admin. You can find all about how to do it on this awesome tutorial. This way you will have the freedom of creating new users from the Django admin with ease.

10. TDD - Test Driven Development

Test driven development is where you write your tests even before you code a feature. The idea is that you will think of the test case, write a test and write the function that will make the test pass. It is certainly not easy to always follow this and getting started with it can feel overwhelming. But for the long run, it'll help you save so much hours of pain and misery for fixing unwanted bugs that happen in production.

I would suggest checking out testdriven.io for getting up and running with TDD [and Docker]. I can vouch that their paid courses are worth every penny without having any affiliation with them.

11. Starter Templates and Style Guides

There are a number of things that always need to be done when starting a new project which are always the same. There are a some fantastic starter templates that can help jumpstart your Django projects. I'm listing a few of them below -

When the project keeps getting larger and multiple developers working on the same project [even while working alone], it is possible to end up with different style codes in the same codebase. It's better to follow a specific style guide throughout all of the codebase so it's easier to read, modify and maintain in the long run. It's also easier to get new developers onboard with the project with a specific style guide. A few of the popular styles guides -

Django Best Practices
Django Styleguide by HackSoftware which also comes with an example project
django-api-domains
Two Scoops of Django 3.x: Best Practices for the Django Framework