Tdarr
patreon
v2 Features!
🕑 Added 2020-12-04 07:50:29 +0000 UTCHello!
The 16 days since my last update have gone by far too quickly but I have some exciting info to go through. Last time, I mentioned "this work may not make much practical difference" but that is now false! There have been some big changes to the scope of the project:
1. Modularisation, nodes and horizontal scaling.
The old version of Tdarr is 'vertically scalable': you can increase the power of your hardware and scale up the number of workers to make the most of it. There have been some makeshift ways to have multiple Tdarr instances running at the same time but soon you'll be able to have all your Tdarr-capable devices working on the same page (i.e. 'horizontal scaling'). While modularising the client and server, I decided to take it a step further and modularise the worker handlers, turning them into independent network-based nodes. Each node appears on the Tdarr dashboard and has its own worker controls. Media files are not sent between server and nodes so nodes will need to have the same relative library mapping as the server.
2. Worker plugin execution
In the old version, plugins are executed in the main server thread. This has become a problem as people have been increasing the complexity of plugins and running all sorts of processes in them which hold up the server. Plugins are now executed in workers so they won't hold up the server or node. Plugins are sent over the network to workers so as soon as you update a plugin on the server, nodes will get the latest version.
3 . ARM device support.
The new node-based system will at the very least enable nodes to be run on ARM devices. Having the server running on an ARM device is dependent on getting ARM versions of all four file scanner binaries (if possible). I'll need to look into this but I now have a Raspberry Pi so can test things easily.
4. Hardware-based workers.
There are now four types of workers: Transcode CPU/GPU and Health Check CPU/GPU. Workers will check the encode commands and reject/accept the task respective of the hardware used. GPU tasks typically have low CPU usage, so this type of system allows maximum hardware utilisation.
5. Cache staging and review.
One of the main gripes of the previous system is the back and forth transcoding/copying of files between the library and cache. Now, all plugin transcoding loops are performed in the cache. Upon completion, you can reject/accept the new file (there'll be an auto-accept switch). If a plugin fails, the last completed cache file (if any) will be available to accept/reject/retry.
So, there you have it, a new Parallel Encoding Network Interfacing System.
Here is the basic architecture from top to bottom:
Client
Server
Node Relay
Node
Worker
Sub-worker
This system introduces all sorts of new challenges so it will take some time to iron out the bugs and make sure each component is communicating the way it should be. I'm planning on having at least a basic preview version (on Windows) available by Sunday next week. The only thing left for me to implement is the accept/reject/retry system and then it will be a cycle of bug fixing, optimisation and review.
Thanks for the support!
HGG
Comments
Tdarr
Some useful space heaters for the Winter :)
Tdarr
Yeah excited to see how it turns out, although Raspberry Pis can’t be good for much more than remuxing haha.
Andy Womsley
Much awesomeness. I have machines of varying capabilities all over the place, now I'll be able to distribute the transcode load around the ranch
James Smith
Oh, fantastic. This sounds *excellent*. The distributed worker model is perfect for the type of work Tdarr does, and being able to distribute work across under-utilised raspberries pi in my house will be great :D
