At some point in your application’s lifetime, someone will want to add data that is not part of your data model. You’ll be faced with a choice. Do you add a weird column to your database, or is there a better way?
Adding a column to your application to handle this request is not a bad thing if you’re a new company. In the early days of a company, you must move quickly to keep up with customer demands. Sometimes you’ll be in need of a quick short-term solution, and this will work “good enough”. I’ve done this plenty of times myself.
However, when this happens, in the back of your mind you should be ready for the next customer to ask for the same thing for their specific data. It’s like having cockroaches… When you see one, that means there are ten more that you don’t see behind your walls. Like cockroaches, if one customer is requesting this functionality, you should anticipate more requests on the way.
The nice thing is that if you handle the initial request as a new database column, you can always go back in the future and “migrate” that column and data into a new Custom Field when you have the time to build this system out. In fact, that’s what I did recently with a start-up company that I’m working on. We have an “Organization” model where we added social links to the database as columns (such as twitter_url). Now that we have a robust Custom Field system, I will migrate that data into our CustomField system, and drop those database columns.
Your company won’t be new and small forever! As you grow and get more customers, you need to think about the long term. The long term solution here is to have a Custom Field System.
What Does a Custom Field System Look Like?
I have worked on a handful of applications that have built Custom Field systems into their applications. Now that I have seen a few, there are things that all of them have in common. Here is a screenshot of what it might look like to add a new custom field:
This is a screenshot of the UI where someone would add, remove, or modify custom fields for this application. As you can see, the fields have the following information:
- A “type” that describes the data type that the field will store. This can be a primitive, such as: integer, string, text, etc. Or, this can be rich data such as: email address, phone number, etc. This is a design decision you will have to decide.
- Each Custom Field must have a name that is unique. I would recommend also storing a separate field on your model as a way to refer to the field through Liquid programmatically (similar to a permalink column that doesn’t change).
- Something that is not shown in this image, but that is important is that the CustomField must belong to something in your application. This will be a Polymorphic relationship. For example, the fields in the picture for “Phone Extension” and “Primary Contact Number” would likely belong to the user model.
Think of Custom Fields as like a database column, but able to be added, removed, or modified on the fly by configuration people. Going back to the fields in the screenshot above, let’s say that we had a User model that had an ID, email, and name columns. We are essentially saying that we are adding two additional fields to this model to store a Phone extension and a Primary Contact Number.
In the systems that I’ve seen and worked with, Custom Fields are typically separated by the Customer. For example, if Customer A is working with volunteers, then it is probably highly valuable to be able to store phone numbers in our system. If Customer B is working with online video gamers, then it is probably more relevant for them to store screen names as their way of contacting people. But that’s the beauty of a Custom Field system. Each customer can decide for themselves what is important, and create the fields that they need.
In Custom Field systems, you typically want to enforce that the data type should never change. Or rather, once there are values, it can be extremely challenging to change. Thinking through this, let’s say that you have a CustomField created to store strings (allowing characters such as a’, ‘b’, ‘c’, etc). Then you decide that you made a mistake and want to change the value to store integers (1,2,3, etc). How are you going to migrate the data? There are some cases where it might be possible, and others where you will definitely have to use some creativity. In the system I've worked with, we simply don’t allow the user to do this.
Custom Field Values
Now that you have your Custom Fields Setup, what’s next? You have basically added on to your database schema without touching your database! The next step is to allow your users to enter data for these new “virtual” columns. Here is an example of what this may look like in your UI:
The UI must render the field depending on the data type. For example, if you have a Date field, you must give the user a way to enter a date value (with a date picker).
Use a text database column to store the data the user enters into the field, and then simply cast the value to the expected data type when reading the data. In some cases this won’t work if your data type is too unique. However, in my experience, this approach does work well the majority of the time.
Validation
When you’re setting up your Custom Field, this is also a good place to add validations to ensure the data you’re collecting is correct. I’ve worked in systems that have two unique approaches toward solving this problem.
The first method of validating is to keep your data types to the bare minimum (primitives), such as “String”. Let’s say that you were trying to model a phone number. The way you would achieve this is by adding individual validations to your field to define what a valid phone number is. As a basic example, you might add a Regex validation to ensure you have ten numerical digits.
The second method of validation is to use “rich” data types with built in validations. For example, if you allow your users to select “phone number” as a data type, you can add built-in validations. The nice thing about this method to solving the problem is that it can be a little more user friendly in the front end when your customers don’t fully understand validations. However, the drawback could be that they want to remove validations that are built in.
A hybrid approach to the validation problem also works well. You can have rich data types, but also allow the user to add or remove additional validations.
You should treat Custom Fields the same way as any other data attribute of your models and validate them the same way. In a Ruby on Rails application, you want to validate your Custom Field Value data at the same time the model that is being saved is validating its own data. This way, you can handle when things are invalid in the same way.
When to Use a Custom Field Vs. a Database Column
One of the problems that happens with having a Custom Field system is that you will constantly need to decide what should be a Custom Field and what should be an actual database column?
Honestly, this shouldn’t be an extremely tough question because it is a highly reversible decision. If you decide to make it a database column and you were wrong, it’s easy to fix. If you decide to make it a Custom Field and you were wrong, it’s just as easy to fix later on.
With that said, here are a couple things to consider when making this decision.
First, if you need to access the data through code, I would always recommend using a database column. Things get extremely weird and fragile when you start referencing data in your codebase (Trust me. I’ve done this before.) Going down this path will likely lead to other fragile code in your codebase, such as if statements to check what customer your code is processing (since most Custom Fields will be scoped by customer). If you can anticipate using the field for more than just reading it, then I would go with a database column or another solution.
Another way to see if you’ve made the right choice in retrospect is to look at patterns in your customer’s data. Does every customer you have always add a “phone number” field to the user model? If that’s the case, then that is probably an indication that this should be a real database column.
The data can also speak in reverse as well. Do you have columns on your models that only a couple of your customers are using? If that’s the case, that is a good candidate for moving to Custom Fields. The great advantage here is that you’re not cluttering the UI for every customer at the request of a small number of customers with the need for those fields. You’ve got a Custom Field system, so use it!
Custom Field Final Thoughts
The first building block of our configuration tool belt is Custom Fields. A good Custom Field system can save you from cluttering your codebase with customer-specific fields that no one else uses.
This is laying the foundation for storing any kind of data that your customer could require, and we will build on this foundation with the next 3 building blocks. Next up, let’s talk about building a Condition System.