Bullet Proofing Django Models

For a better reading experience, check out this article on my website.

We recently added a bank account like functionality into one of our products. During the development we encountered some textbook problems and I thought it can be a good opportunity to go over some of the patterns we use in our Django models.

This article was written in the order in which we usually address new problems:

  1. Define the business requirements.
  2. Write down a naive implementation and model definition.
  3. Challenge the solution.
  4. Refine and repeat.

A Bank Account

Image for post
Image for post
  • Each user can have only one account but not every user must have one.
  • The user can deposit and withdraw from the account up to a certain amount.
  • The account balance cannot be negative.
  • There is a max limit to the user’s account balance.
  • The total amount of all balances in the app cannot exceed a certain amount.
  • There must be a record for every action on the account.
  • Actions on the account can be executed by the user from either the mobile app or the web interface and by support personnel from the admin interface.

Now that we have the business requirements we can start with a model definition.

Let’s break it down:

  • We use two unique identifiers — A private identifier which is an auto generated number (id) and a public id which is a uuid (uid). It’s a good idea to keep enumerators private — they expose important information about our data such as how many accounts we have and we don’t want that.
  • We use OneToOneField for the user. It’s like a ForeignKey but with a unique constraint. This ensures a user cannot have more than one account.
  • We set on_delete=models.PROTECT. Starting with Django 2.0 this argument will be mandatory. The default is CASCADE — when the user is deleted the related account is deleted as well. In our case this doesn’t make sense — imagine the bank “deleting your money” when you close an account. Setting on_delete=models.PROTECT will raise an IntegrityError when attempting to delete a user with an account.
  • You probably noticed that the code is very… “vertical”. This is not just because Medium has such poor support for source code — we write like that because it makes git diffs look nicer.

Now that we have an account model we can create a model to log actions made to the account:

What do we have here?

  • Each record will hold a reference to the associated balance and the delta amount. A deposit of 100$ will have a delta of 100$, and a withdrawal of 50$ will have a delta of -50$. This way we can sum the deltas of all actions made to an account and get the current balance. This is important for validating our calculated balance.
  • We follow the same pattern of adding two identifiers — a private and a public one. The difference here is that reference numbers for actions are often used by users and support personnel to identify a specific action over the phone or in emails. A uuid is not user friendly — it’s very long and it’s not something users are used to see. I found a nice implementation of user-friendly ID’s in django-invoice.
  • Two of the fields are only relevant for one type of action, deposit — reference and reference type. There are a lot of ways to tackle this issue — table inheritance and down casting, JSON fields, table polymorphism and the list of overly complicated solutions goes on. In our case we are going to use a sparse table.

Note about the design: Maintaining calculated fields in the model is usually bad design. Calculated fields such as the account’s balance should be avoided whenever possible.

However, in our “real life“ implementation there are additional action types and thousands of actions on each account — we treat calculated attribute as an optimization. Maintaining state poses some interesting challenges and we thought it can serve the purpose of this post so we decided to present it as well.

Challenges

We have three client applications we need to support:

  • Mobile app — use an API interface to manage the account.
  • Web client — can use either an API interface (if we have some sort of SPA) or good old server side rendering with Django forms.
  • Admin interface — uses Django’s admin module with Django forms.

Our motivation is to keep things DRY and self contained as possible.

We have two types of validations hiding in the business requirements:

Input validation such as “amount must be between X and Y”, “balance cannot exceed Z”, etc — these types of validation are well supported by Django and can usually be expressed as database constraints or django validations.

The second validation is a bit more complicated. We need to ensure the total amount of all balances in the entire system does not exceed a certain amount. This forces us to validate an instance against all other instances of the model.

Race conditions are a very common issue in distributed systems and ever more so in models that maintain state such as bank account (you can read more about race conditions in Wikipedia).

To illustrate the problem consider an account with a balance of 100$. The user connects from two different devices at the exact same time and issue a withdraw of 100$. Since the two actions were executed at the exact same time it is possible that both of them get a current balance of 100$. Given that both session see sufficient balance they will both get approved and update the new balance to 0$. The user withdrawn a total of 200$ and the current balance is now 0$ — we have a race condition and we are down 100$.

The log serves two purposes:

  • Log and Audit — Information about historical actions — dates, amounts, users etc.
  • Check Consistency — We maintain state in the model so we want to be able to validate the calculated balance by aggregating the action deltas.

The history records must be 100% immutable.

The Naive Implementation

Let’s start with a naive implementation of deposit (this is not a good implementation):

And let’s add a simple endpoint for it using DRF @api_view:

So what is the problem?

  1. Locking the account — An instance cannot lock itself because it had already been fetched. We gave up control over the locking and fetching so we have to trust the caller to properly obtain a lock — this is very bad design. Don’t take my word for it, let’s take a glimpse at Django’s design philosophy:

Loose coupling
A fundamental goal of Django’s stack is loose coupling and tight cohesion. The various layers of the framework shouldn’t “know” about each other unless absolutely necessary.

So is it really the business of our API, forms and django admin to fetch the account for us and obtain a proper lock? I think not.

2. Validation — The account has to validate itself against all other accounts — this just feels awkward.

A Better Approach

We need to hook into the process before the account is fetched (to obtain a lock) and in a place where it makes sense to validate and process more than one account.

Let’s start with a function to create an Action instance and write it as a classmethod:

What do we have here:

  • We used a classmethod that accepts all necessary data to validate and create the new instance. By *not* using the default manager’s create function (Action.objects.create) we encapsulate all the business logic in the creation process.
  • We easily introduced custom validation and raise proper ValidationError.
  • We accept the creation time as an argument. That might seem a bit strange at first glance — why not use the built in auto_time_add? For starters It’s much easier to test with predictable values. Second, as we are going to see in just a bit, we can make sure the modified time of the account is exactly the same as the action created time.

Before moving over to the implementation of the account methods let’s define custom exceptions for our Account module:

We define a base Error class that inherits from Exception. This is something we found very useful and we use it a lot. A base error class allows us to catch all errors coming from a certain module:

A similar pattern can be found in the popular requests package.

Let’s implement the method to create a new Account:

Pretty straight forward — create the instance, create the action and return them both.

Notice how we accept asof here as well — modified, created and the action creation time are all equal — you cant do that with auto_add and auto_add_now.

Now to the business logic:

We can start to see the pattern here:

  1. Acquire a lock on the account using select_for_update. This will lock the account row in the database and make sure no one can update the account instance until the transaction is completed (either committed or rolled-back).
  2. Perform validation checks and raise proper exceptions — raising an exception will cause the transaction to rollback.
  3. If all the validations passed update the state (current balance), set the modification time, save the instance and create the log (action).

So how does the model hold up to our challenges?

  • Multiple Platforms and Validation We encapsulated all of our business logic including input and system wide validation inside the model method so consumers such as API, admin action or forms only need to handle exceptions and serialization / UI.
  • Atomicity — Each method obtains its own lock so there is no risk of race condition.
  • Logging / History — We created an action model and made sure each function registers the proper action.

Profit!

Testing

Our app will be incomplete without proper tests. I previously wrote about class based testings — we are going to take a slightly different approach but still have a base class with utility functions:

To make testing easier to write we use utility functions to reduce the boilerplate of specifying the user, the account etc each time by providing default values and operating on self.account.

Lets use our base class to write some tests:

Final Words

The classmethod approach has proved itself in our development for quite some time now. We found that it provides the necessary flexibility, readability and testability with very little overhead.

In this article we presented two common issues we encounter frequently — validation and concurrency. This method can be extended to handle access control (permissions) and caching (we have total control over the fetch, remember?), performance optimization (use select_related and update_fields…), audit and monitoring and additional business logic.

We usually support several interfaces for each model — admin interface for support, API for mobile and SPA clients, and a dashboard. Encapsulating the business logic inside the model reduced the amount of code duplication and required tests which leads to overall quality code that is easy to maintain.

In a follow up post I (might) present the admin interface for this model with some neat tricks (such as custom actions, intermediate pages etc) and possibly an RPC implementation using DRF to interact with the account as an API.

Written by

Full Stack Developer, Team Leader, Independent. More from me at https://hakibenita.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store