(Amiberry powered) Lightwave render farm, pt 2

By | 2020-05-30
0 Comment

To make this crazy project a reality, we will need a few things. I’ve made a list of what I used and how I did it, but it may not be the best solution for everyone out there. Feel free to adapt things according to your needs and budget, of course. And in the case you find a better solution to some of the items listed here, I’d be happy to hear from you!

Hardware Requirements

Let’s start with the list of hardware items we’ll be using:

  • An Amiga computer (real or emulated) that will be running Lightwave. We will be loading the Scenes and starting the job from here, which will be distributed to our render farm afterwards. We will be calling this machine the “Master”. I’m going to use my real Amiga for this task, just for the fun factor of it.
  • Some Raspberry Pi boards (or any other board capable of running some UAE flavor really) for the render farm. You can use as many as you want, and it will work even with just one. Obviously, the faster they are, the better. We will be calling these devices “Slaves”. I’ve got 6 Raspberry Pi devices lying around, so I’ll put them all to good use in a cluster enclosure.
  • SD cards for all those Raspberry Pi boards. We won’t need a lot of space on them, since the output frames will be saved directly in the shared location. I usually go for 16 or 32GB SanDisk Ultras.
  • A common storage location to hold the 3D content we want to use, as well as the output rendered frames. This location should be shared over the network so that all the devices can access it. It can be located on one of the existing machines, on a separate NAS or any other device, as long as it’s accessible over the network from all. After many tests with different protocols, I’ve chosen Samba as the most reliable and easy to use sharing method here. More details on this later.
  • Ethernet switch or WiFi router, to get all the above devices connected to the same network. An ethernet switch makes it easy to get everything running quickly, since you just plug in the cable and you’re done (although you end up having lots of cables around). Wifi needs some extra work per device (selecting your Access Point, entering the passphrase), but it gives you more space freedom. You can also mix the two (e.g. real Amiga on Ethernet, Raspberry Pi on Wifi), as long as they are in the same network.
  • Ethernet cables for your devices, if you choose to use an Ethernet switch.
  • Power supplies for the devices. Kind of obvious, but you may want to plan ahead if you have multiple Raspberry Pis – their official PSUs take up some space, so make sure you have enough free sockets! An alternative for this would be to use Power Over Ethernet, but it requires some different pieces of hardware (a Switch that supports it, a special HAT for the RPIs), so I won’t be covering that here.
  • (Optional): a Router for easy DHCP, name resolution and routing. If you already have one at home, you’re probably set. But if you’re planning on making this farm portable, you might want to think about this part. An alternative would be to use static IP addresses, but that involves a bit more manual work, and is not covered in this guide.

Software Requirements

And now let’s take a look at the software requirements:

  • An image of the latest Raspberry Pi OS – You could also use other alternatives (such as Manjaro Linux), but those are not covered in this guide.
  • The latest Amiberry version. We’ll just grab the latest binaries from the Github Releases page, to save ourselves the effort of compiling. If you chose another distro, you will probably need to compile Amiberry from source instead. Just follow the instructions in the Readme.
  • Optional: Kickstart ROM files. Amiberry, like any other UAE implementation, requires the actual Amiga Kickstart ROM files to function. If you have the Amiga Forever package, you can use those directly. An alternative AROS ROM is included in the emulator, and will be used automatically, if you don’t have them. For our purposes, even the AROS version will work just fine.
  • A directory containing all the Lightwave 3D content you want to render. The convention used is that in this directory (let’s call it “Projects” from now on), there should be a directory named “3D” which includes all the content (e.g. Scenes, Objects, Images), one named “command” which should be empty (this will hold the generated job files), and optionally some directory for saving the output frames after rendering. The name of the later depends on your Scenes, as output directory names and format are configured there.
  • Samba sharing between the the shared content location mentioned above and the “Slaves” and “Master” devices. For example, if the shared location is on a NAS, you will need to enable SMB Sharing for it. If it’s on a Windows machine, you will need to enable Windows File Sharing for it. If it’s on a Linux machine, you will need to install and configure Samba accordingly (more details on that below).
  • The “RenderNode” special boot environment. This is just a minimal AROS installation that I created, which includes Lightwave’s “ScreamerNet” tool and some custom scripts to automate things, so we can limit user-input to the minimum. After all, we want this thing to run automatically, don’t we? We will use the “RenderNode” as a virtual HDD in Amiberry.

Configuration

Got everything? Great! Let’s start by configuring the Raspberry Pi devices first. We will do these steps on all of them.

  • First of all, let’s update the system. Open a console and type:
sudo apt update && sudo apt upgrade -y

This will probably take a while, if it’s the first time you’re running it. Reboot if necessary, when it ends.

  • Now let’s enable SSH, so we can easily connect to them remotely and control them, without having to use a keyboard on each device. Open the raspi-config tool to do this easily:
sudo raspi-config

Then go to Interfacing Options -> SSH and select to Enable the SSH Server.

  • Make sure that the system uses the FKMS driver, otherwise Amiberry will not be able to use hardware acceleration through DispmanX. You can do this through raspi-config as well. Go to Advanced Options -> GL Driver and select the “GL (Fake KMS) OpenGL desktop driver with fake KMS” option. Reboot if necessary.
  • Install all the necessary system requirements to run Amiberry (as found in the Readme). If you want to be running Amiberry from the full graphical Desktop (which is the easiest approach as it requires the least effort), then you just need these:
sudo apt install cmake libsdl2-2.0-0 libsdl2-ttf-2.0-0 libsdl2-image-2.0-0 flac mpg123 libmpeg2-4 libserialport0 libportmidi0

(Check the Readme linked above for other options, such as compiling Amiberry from source)

  • We’ll need to assign unique hostnames to our Raspberry Pi boards, with a common Domain name. This is optional, but it will make it easier to manage them, since we won’t have to remember a list of IPs. I named mine rpi3-1.local, rpi3-2.local, rpi3-3.local etc. You can be more creative if you want. Here’s a guide on how you can achieve this, step by step.

At this point, you’ll have to decide where your shared content directory will reside. Like I mentioned before, you can use different configurations to achieve this (e.g. a NAS, a Windows server, a Linux server, even one of the Raspberry Pis). Whatever choice you make, you’ll have to ensure that it has a unique hostname (or IP Address) that can be reached from all the “Slaves” and your “Master” computer. In the next part, I will provide a few example configurations to get you started in the right direction, and hopefully make your life easier. Following that, we’ll proceed with mounting the shared directory from all the “Slaves” and your “Master” and make sure it all works!