What's new

Project: Compression is for pussies

So, I want to build a media server. I want this server to run on my LAN and allow me to stream my movies to my devices. I do want to allow this to also send movies over the internet if I'm at a friend's house (This would be of course have a secure log in page to keep the server secure). The initial design included a Raspberry Pi 3 Model B with a few 1TB pi-drives in a RAID 0 for a large storage solution.

(Notice how I use Gb and MB in the upcoming paragraph. The B stands for byte and the b is for bit. And keep a mind that it takes 8Mb to make 1MB since it takes 8 bits per byte and 1024MB to make 1GB)

I then was able to get an hour or two to bounce some ideas off Hades and we talked about all the different bottlenecks and requirements would be needed for 4 simultaneous 4K HDR movie streams at one time. We had it figured that I would need a 10Gb switch to increase the data transfer speeds of my network since one 4K stream has a 60MB/s bitrate (Scroll down for Graphs). Now I want to allow 4 4K streams at 60MB/s meaning I would need a minimum 240 MB/s for the movies alone not taking other network traffic into effect. Which by the way if you didn't know is 1.88Gb/s. Most home routers and devices have 1Gb/s. So I would easily be stealing all the bandwith and I might as well get a 10Gb switch cause I need to future proof for 8K. Plus you can't tell me a 10Gb switch isn't making you get hard. Also, keep in mind that for streaming across the internet I would need a 2Gb upload speed to allow for the 4 streams to push at once outgoing and .5Gb download speed for each client. Now, these numbers could drop with compression but look at the title bro.

Now that we have the network requirements over let's look at the storage solutions Hades and I pondered over. Now since I'm trying to push 4 Streams at once spinning drives would bottleneck performance due to them not being able to keep up with 240MB/s random read speeds. So an SSD could easily squash this problem but, cost then comes into effect. A 120GB SSD is around $50 USD. A 1TB HDD is around $50 USD. So Hades and I came up with two options.

Option A
Buy 2 4TB HDD's and put them in RAID 1 to increase random read performance then put a 120GB SSD in between as a caching device to fill a buffer then feed that to the media server. Now, this can be achieved with some custom scripts that would be in charge of filing the movies and transferring as needed.

Option B
Buy a 4TB SSD and just read directly.

Option A would cost me roughly $450 USD while Option B would cost me $1500 USD.
Option B would futureproof the fuck out of the system for 4K and possibly hiccupping at 8K. So futureproofing comes at a price in storage solutions till SSD prices drop.

The next step is the server. The server itself wouldn't need much in processing since all the rendering and stuff is done on the client. It is more on the memory side since you would need a pretty decent buffer to allow 4 different 4K streams which Hades and I chalked up to 12GB's of ram. The server would also need 4 1Gb/s Networking cards for each individual stream to allow maximum performance on the switch.

Now, I know a lot of people are saying why do you need 4 4K streams going at one time. In all actuality, it will probably only run 1 - 2 1080P streams at one time, but I want to give this machine a good amount of power for bragging rights and also to be given the possibility for 4K streams. This machine will be used by My Roommate and me who share one TV so that's one stream and probably my grandmother so she can have all the movies she can dream to view.
Anyway, let me know what you think. I'm open for suggestions and criticisms. I will also be posting updates for my own records.

TODO list:
- Figure Hardware costs
- Write backbone for server (Scripts)
- Build and Setup
 
Last edited:

HADES

Vanguard
Staff member
DG | Developer
DG | Staff
DG | Admin
DG | Veteran
DG | Member
Just a clarification... the server step does not use a Raspberry Pi. In this server scenario, a PC is used. Also there is no need for 10Gb/s switch.

The basic server requirements are simple:
1) A gigabit network card with 4 ports
2) 16 GB of RAM

The host requirements are also simple:
1) A gigabit network card
2) 4k capable display

Here is a general diagram of how the minimum setup looks:
Setup1.jpg



The Media Server with 2 HDDs in RAID 1:

Setup2.jpg


The Media Server with 4 HDDs in RAID 10:

Setup3.jpg


This last scenario is probably the best as it delivers the most speed from the spinning disks. The 14GB cache should be sufficient to provide smooth streaming experience and help the spinning disks with random reads with 4 x 4k streams. The cache can be increased if the spinning disks can't keep up.

The RAID setup can be done with a cheap RAID card, or even software RAID.
 
Just a clarification... the server step does not use a Raspberry Pi. In this server scenario, a PC is used. Also there is no need for 10Gb/s switch.

The basic server requirements are simple:
1) A gigabit network card with 4 ports
2) 16 GB of RAM

The host requirements are also simple:
1) A gigabit network card
2) 4k capable display

Here is a general diagram of how the minimum setup looks:
View attachment 97


The Media Server with 2 HDDs in RAID 1:

View attachment 98

The Media Server with 4 HDDs in RAID 10:

View attachment 99

This last scenario is probably the best as it delivers the most speed from the spinning disks. The 14GB cache should be sufficient to provide smooth streaming experience and help the spinning disks with random reads with 4 x 4k streams. The cache can be increased if the spinning disks can't keep up.

The RAID setup can be done with a cheap RAID card, or even software RAID.
Thanks for the graphs. Really helps me visualize the whole project.
 
Top