Use a RAM disk to store temporary HLS video fragment files of your live-stream server

What exactly is a RAM disk?
Why would I want to use a RAM disk?
How large should the RAM disk be?
How to create a RAM disk

I’ve posted several articles in the past, most noteably this one, on the subject of live-streaming, so it shouldn’t be a real big surprise that I’m doing another post on it. A RAM disk can enhance your live-stream server, while at the same time increasing the lifetime of your storage device. But first, let’s ask an A.I. how it would describe a RAM disk in such a way that anybody will be able to understand it.

What exactly is a RAM disk?

“A RAM disk is a section of your computer’s memory (RAM) that is treated like a hard drive. It allows you to store and access files at extremely high speeds because RAM is much faster than traditional storage like hard drives or SSDs. However, since RAM is volatile, any data stored on a RAM disk is lost when the computer is turned off or rebooted. RAM disks are useful for temporary storage of data that needs to be accessed quickly but doesn’t need to be saved permanently.“

I couldn’t have put it any better myself! Thank you A.I.! And also thanks for confirming, once again, that A.I. is a super awesome tool for people who are as lazy as I am. Perhaps I should have this A.I. write the entire posts in the future!? Who is to say that I didn’t do so already? 🙂 I’m kidding.

I would like to mention that using a RAM disk in combination with your HLS live-stream server, will never result in any negative impact on the performance of your live video feeds. In fact, the opposite is more likely, as it can only improve on the performance of the live-stream, since RAM is so much faster than any storage device. I hope everybody is now caught up on what a RAM disk is. Now let’s find out if you will want to be using one, or not. I’m guessing that you will!

Why would I want to use a RAM disk?

Whenever your live-stream server is actively being used, it will continuously write temporary video fragment files (and optionally key files) to a location on your hard disk that you’ve specified in the nginx config file. These temporary files compose the HLS stream your server generates. After a short time, these temporary files are deleted again, to make room for new ones. This continuous activity keeps your hard drive busy a lot, and this causes wear and tear. Especially SSD drives don’t like being used (or abused?) like this. And their useable lifetime will definitely reflect this, negatively. In other words, if your live-stream server is used a lot and often, then you can bet your sweet bippy that the disk will have a much shorter life compared to drives that aren’t used like this.

If you’re using a VPS that you rent from some provider, this is not something you really need to worry about, since it is their responsibility to assure reliable storage for your server.
In case you self-host a live-stream server that runs on your own hardware, on a machine at home or the office, then you should seriously consider using a RAM disk. Even more so when you’re using a folder on the same drive as where the operating system runs on, because when this drive breaks down, you’ll also lose the operating system, so the entire server will basically be lost.
Did I manage to persuade you to consider doing this already? I sure hope so, because I can’t wait to get started with the actual tutorial. I promise you, it’s super easy to follow, and we’ll be done within a matter of minutes.

How large should the RAM disk be?

This depends on a couple of factors. You’re going to need to decide on a size on your own, by using your brain. I know, I know, this is a lot to ask, but trust me, it’s all going to be fine. This is where a third of the readers closes their tab to this post. You’re still reading this? Good for you!
The first step would be to figure out how much RAM your server has. If it is 4 GB of RAM or less, then I suggest you forget the whole idea of a RAM disk. If it’s between 8 and 12 GB RAM, I recommend to first monitor the server’s memory usage closely for a while, to determine if it even has any RAM to spare. When we get above 12 GB of RAM, the changes that it can spare some RAM obviously increase. Once we get at 16 GB of RAM or above, we can be pretty certain that borrowing some RAM won’t impact the server in any way.

Since we are borrowing precious RAM space, you’ll want to choose a number that is not too large, so we don’t take away too much RAM from the server for it to operate as normal. At the same time, we need to make it large enough so that it’s able to store all the video fragment files our live-stream server will generate. I can not tell you how big this should be, because it depends on too many variables. To name some variables, it depends on: How many live-streams are configured? How long, and thus how large, are the individual fragments? In case you’re streaming adaptive bitrate, how many of them? Once you’ve decided on the size you want, we can finally create a RAM disk.

In this example, I was using a server with 16 GB of RAM, with enough to spare. I know for a fact that the size of the folder that stores the HLS fragments uses 200 MB at max. A reasonable choice would be a size of 512 MB for the RAM disk. But to make absolutely sure it will be large enough, and since I can spare it, I went for a RAM disk of 1 GB, leaving 15 GB for the server to operate. I’m assuming that 1 GB will be plenty big enough for you as well, but like I said earlier, use your brain here to decide due to all the possible variables.

How to create a RAM disk

Stop Nginx entirely

Stop all the live-streams to the server, and then stop Nginx as well. Speaking of Nginx, this is the app that will be using the RAM disk primarily, and we know that Nginx runs as the user www-data with the same group ID. Obviously, we want Nginx to have full read and write access to the RAM disk, so we’ll want to make sure it has the correct rights for this in place. There are several ways to achieve this, and I believe the method I’m going to show you is the easiest one. More on this further down when we get to it. Let’s not get ahead of ourselves and start by stopping Nginx.

sudo systemctl stop nginx

You may need to take a quick peek at the config files for Nginx to figure out what folder it was using to store the HLS files. We can do this easily by entering this:

"grep -oP '(?<=hls_path\s)[^;]+' /etc/nginx/nginx.conf

This will return the value that you are using to store the HLS files. In this example, it’s /mnt/livestream/hls. We want the hls folder to be stored on the RAM disk in the future, so the folder we are using would be: /mnt/livestream.

Clear out, or create the folder for our mounting point

If you happened to have followed this tutorial to the letter, you are using the same folder as shown in the example. When live-streaming, this folder should show several subfolders in it, like “hls”, “dash” and “keys”. We want all these sub folders to be stored on the RAM drive in the future, but not any other folders. We don’t want to fill up our RAM disk with unnecessary files. So make sure that you move any folders that you perhaps at some point added to /mnt/livestream out of there, in to a different location. Or better yet, let’s just rename the folder and create a new one for the RAM disk, and a new one for recordings. This way, any files you may have stored in there are untouched, and we have a clean, empty folder for mounting the RAM disk.

sudo mv /mnt/livestream /mnt/livestream_bckp && sudo mkdir /mnt/recordings /mnt/livestream && sudo chmod 777 /mnt/livestream && sudo chown -R www-data: /mnt/recordings /mnt/livestream

Of course, it is entirely up to you to use a different folder for the RAM disk. But I don’t want to hassle with editing the Nginx config files right now, so I choose to create the RAM disk in the exactly same location as specified in the nginx.conf file. But, feel free to pick any location you want, just make sure to create that folder before continuing, because the folder location must exist, or else it’s not going to work. Also use your brain and not blindly copy/paste everything from this tutorial, but edit them first so they will suit your setting.

Change the “record_path” setting

So what we also want to do, is change the location that Nginx uses whenever you decide to record your live-streams on the server. These would fill up the RAM drive fast, and it’s not safe to store them there. You need to be aware that any file that you will store on the RAM disk will be lost whenever the server reboots. Further down this tutorial, I will show you a method that makes it seem as if these files are stored persistently, but that is simply a trick. Besides, you don’t want to store any recordings on the RAM disk anyways, for several obvious reasons. The following command will change the “record_path” setting in the nginx.conf file from /mnt/livestream/recordings to /mnt/recordings. This will only work if you followed the tutorial. If you haven’t, just edit /etc/nginx/nginx.conf and make sure the record_path is set to a location that is no longer inside the folder for the RAM disk.

sudo sed -i 's|record_path /mnt/livestream/recordings|record_path /mnt/recordings|' /etc/nginx/nginx.conf

Mount up!

Later on, I’m going to show you how to mount the RAM disk automatically at boot time. Which in my opinion is the preferred way of doing this. But for now, we can do a test-run by entering the mount command for creating a RAM disk. Please note that I specified the size using size=1024m, and that is something you need to edit to whatever you’ve chosen it to be. 

sudo mount -t tmpfs -o size=1024m,uid=www-data,gid=www-data ramdisk /mnt/livestream

In case you use a desktop environment on your server, and you want to be able to see the RAM disk as a drive in your GUI, you need to add “x-gvfs-show“, and it should be placed directly after gid=www-data. sudo mount -t tmpfs -o size=1024m,uid=www-data,gid=www-data,x-gvfs-show ramdisk /mnt/livestream

Now simply type “mount” to see if it was indeed mounted.

mount

You should see a line that will look familiar to what is shown in this screenshot.

“ls -ld /mt/livestream” should show you the ownership and permission. If you’d like to perform a quick test in orde to confirm that user www-data has write access to the RAM disk, use the command below, that will mask you as user www-data and try to write a test.txt file to the RAM disk.

sudo su www-data -s /bin/bash -c "touch /mnt/livestream/test.txt" && ls -la /mnt/livestream

It should look something like this.

You have now created a RAM disk! See how easy this is? To unmount the RAM disk simply use the umount command as shown below. 

sudo umount /mnt/livestream

Or, you could also do a little bit of testing before you unmount it, if you like.

Do some testing

Using dd, we can do some tests that will show the read/write speeds of the RAM disk device.

sudo dd if=/dev/zero of=/mnt/livestream/zero bs=4k count=100000 && sudo dd if=/mnt/livestream/zero of=/dev/null bs=4k count=100000

You will see the read/write speeds in what it returns. It should be faster when compaired to the same test for the drive. This will perform the same test, but this time on your regular hard drive:

sudo dd if=/dev/zero of=/mnt/zero bs=4k count=100000 && sudo dd if=/mnt/zero of=/dev/null bs=4k count=100000

In my case I had to repeat both these test a couple of times in a row to see the avarage speeds, since the results sometimes varied a lot. But in any case, you’ll see that the speed of the RAM disk is always higher.

Don’t forget the remove the files that dd test just created, since they are pretty big. 

sudo rm /mnt/livestream/zero
sudo rm /mnt/zero

Mount the RAM disk at boot time

This is not a requirement at all, so if you prefer to manually mount the RAM disk each time after a reboot, that’s of course totally fine. In my case, it makes sense to automatically mount it at each boot, since I know that I want to have it in place at all times, for when I spontaniously decide to start streaming for example. First unmount the RAM drive if you haven’t already.

sudo umount /mnt/livestream

Now we’re going to edit /etc/fstab

sudo nano /etc/fstab

And add a new line to it. If you are using a GUI desktop environment, include this: x-gvfs-show

ramdisk   /media/livestream    tmpfs   defaults,size=1G,uid=www-data,gid=www-data,mode=0777  0 0

Save the file, close it, and use mount -a

sudo mount -a

All done! Good job! Enjoy live-streaming in the future with the safe knowledge that your storage device is being saved from wear and tear. Feels a lot better doesn’t it? Was I able to make you happy with this article? If so, leave a thanks in the comments please! Or, continue reading the next part, and then leave a comment.

There is something that I cooked up that will save all the files that are on the RAM disk when the server is going down for a reboot, and it restores them as soon as it is booting up again. This kinda of mimmicks the behaviour of persistent file storage, but obviously it is just a trick. A trick that is not 100% fail-proof, so keep that in mind. In addition, you can also set a cron job that will automate a task that copies all the files from the RAM disk to another location. If this sounds something you also want to do, please continue reading, as I’m going to show you how to achieve this.

You may be wondering at this point why I choose to do this, since the HLS files get automatically deleted when live-streaming has stopped, and the subfolders will also disappear. This eventually leaves the drive empty. So why create a task that saves and restores any files on it? Well, I’ve set up my server in such a way that not only Nginx uses the RAM disk. So there are some files and folders on there that I prefer to have back in place after each reboot. And also with the folder owner and persmissions the same as how I require them to be. So what I did was create a new folder, and use it as some sort of template for the RAM disk by adding the files and folders I want on it.

Make the RAM disk appear as persistant storage device

“A RAM disk is inherently volatile, meaning its contents are lost when the system is powered down or rebooted. However, if you want to preserve certain folders or files on the RAM disk between reboots, you can set up a process to save the contents to persistent storage before a reboot or shutdown and then restore them on the next boot.”

We can create a script that runs during shutdown or reboot to copy the contents of the RAM disk to a persistent location, such as a directory on your regular hard drive. In this example this will be /var/ramdisk/backups.

Script to save RAM disk contents

We’re going to create a tiny script that copies the contents from the RAM disk to a location specified in the script. So be sure to edit it to your wishes. Create a new file /etc/init.d/save_ramdisk.sh

sudo nano /etc/init.d/save_ramdisk.sh

Copy the content below in to the file, and choose your prefered backup location.

# Define the source and destination directories
RAMDISK_DIR="/mnt/livestream"
BACKUP_DIR="/var/ramdisk/backups"

# Create the backup directory if it doesn't exist
mkdir -p $BACKUP_DIR

# Copy the contents of the RAM disk to the backup directory
rsync -a --delete $RAMDISK_DIR/ $BACKUP_DIR/

Explanation: This script uses rsync to copy the contents of the RAM disk to a backup directory. The –delete option ensures that files removed from the RAM disk are also removed from the backup, keeping the backup directory in sync with the RAM disk.

Make the script executable:

sudo chmod +x /etc/init.d/save_ramdisk.sh

Register the script to run at shutdown and reboot:

sudo ln -s /etc/init.d/save_ramdisk.sh /etc/rc0.d/K01save_ramdisk \
sudo ln -s /etc/init.d/save_ramdisk.sh /etc/rc6.d/K01save_ramdisk

Script to restore RAM Disk contents

The second script we’ll create will be /etc/init.d/restore_ramdisk.sh

sudo nano /etc/init.d/restore_ramdisk.sh

Paste the following in to it while changing the backup dir location to your needs.

#!/bin/bash

# Define the source and destination directories
BACKUP_DIR="/var/ramdisk/backups"
RAMDISK_DIR="/mnt/livestream"

# Check if the backup directory exists and is not empty
if [ -d "$BACKUP_DIR" ] && [ "$(ls -A $BACKUP_DIR)" ]; then
    # Copy the contents of the backup directory to the RAM disk
    rsync -a $BACKUP_DIR/ $RAMDISK_DIR/
fi

Explanation: This script checks if the backup directory exists and is not empty, then restores the contents to the RAM disk using rsync.

This script also needs to become executable:

sudo chmod +x /etc/init.d/restore_ramdisk.sh

And to register this script so it will run at boot create a link to it like this:

sudo ln -s /etc/init.d/restore_ramdisk.sh /etc/rc2.d/S01restore_ramdisk

With those in place, it might be a good idea to do a test at this point. We can test the backup and restore process by manually running the scripts and then performing a reboot. After rebooting, check if the folders and files you wanted to preserve have been restored to the RAM disk.

To further ensure data preservation, you could set up a cron job that periodically saves the RAM disk contents while the system is running. This is the last time of me trying to encourage you to continue reading. I promise.

Automate periodic backups

To further ensure data preservation, you could set up a cron job that periodically saves the RAM disk contents while the system is running:

sudo crontab -e

Add a line like this to save the RAM disk contents every hour:

0 * * * * /etc/init.d/save_ramdisk.sh

This setup will allow you to effectively preserve specific folders or files on the RAM disk between reboots, mimicking persistence for a normally volatile storage medium.

Don’t forget to leave a thanks in the comments if you found this page useful!

When I post new content on this website, an automatic notification is sent out to a group of people that subscribed to my website. To become a part of this group is really easy. Enter your email adres below and click on Subscribe. Your address is not shared with any 3rd parties, and it will only be used to send you a notification. Pinky promise.