Challenges¶
Now that we have (hopefully) sold you on the idea of self-hosting, we can discuss some of the challenges that it presents in practice. There are quite a few, but we have been working hard to find solutions for all of the ones you might encounter. In summary, the challenges for self-hosted software are:
- Ability to install software in an isolated manner
- Backing up applications automatically
- Restoring applications from backups automatically
- Incoming connectivity
- Sustainability of applications and platforms
Let's go through each of these issues, and explain how it is that we have solved them.
Software Isolation¶
Applications are complicated things. They require specific support software to be able to run, this could be specific libraries, or external dependencies such as message brokers, databases, queues, schedulers. One of the biggest challenges with easily deploying applications is being able to set everything up. In the primitive old days, installing an application was a task that could take days: the list of support software has to be identified, then the software has to be installed manually on a fresh server, configured appropriately, tested, and then the software itself has to be installed. If any of the support software was configured incorrectly, or if the wrong version of a dependency was mistakenly installed, then nothing would work. If you wanted to install two programs that each depend on something, but on different version of that thing, then well: you were out of luck. If the application that you installed happens to be vulnerable and some bad actor exploits the vulnerability, your entire new server can be hacked.
Today, thanks to big-brain technologies like Linux cgroups, we have a workable solution to that problem. A solution so powerful that the current cloud infrastructure is actually almost fully built on this technology: containerization.
The term containerization is borrowed from the shipping industry. Long story short, shipping goods internationally (and even nationally) was expensive and inefficient because it was difficult to load and unload ships efficiently. When the shipping container was introduced, which has standardized dimensions, shipping became much more effective, because it was possible to stack the containers on top of each other, making the most use of ships, and it was much faster to unload them as they could be unloaded by machines, put on trucks to be transported further without all of the good needing to be repacked.
Software containerization does the same thing. An application, along with all of the dependencies that it needs to run (code and data) is packed into a container, think of it like a ZIP file. This container is interchangeable, it can run on any (Linux) system. Applications packaged in this way run isolated, which means that it is possible to run two applications that require different versions of the same dependency. The isolation also has security benefits: if a single application gets hacked, the attacker has no access to anything else running on that server. With containerization, applications no longer need to be manually installed on a server by a human operator, but rather can be automatically deployed to a whole fleet of servers without human intervention. Because of that ease, software can also be automatically moved between machines to use up computing resources most effectively.
The software that implements this is called Docker, and this is a part of what powers Mosaic. All applications that run on Mosaic run in Docker containers, which means that even decades from now you will be able to run your old applications, as they are fully isolated from the underlying operating system.
Backups and Reliability¶
Computers do not have an infinite lifespan. Drives can fail, corrupting data in the process. Other hardware can fail, rendering a machine unbootable. Sometimes the internet connection fails, making a machine unreachable. But yet, there is nothing worse than an application that you cannot use because the machine running it has died or is unreachable.
One of the ways that we can solve this is by making sure we take a lot of backups of your data. This means that when a machine does fail, you can easily restore it from the backup. To make this an effective strategy, it has to be seamless and it has to be automatic, and that is exactly what we have done. When you run applications with Mosaic, we take what is called a snapshot of the application state every two minutes, and then we send only the changes since the last snapshot to storage in a fully end-to-end encrypted way. This is the optimal strategy for ensuring that if your device gets lost somehow (broken, drowned, burned down), you can rest easy at night knowing you will never lose more than two minutes worth of your data. And because we only store the changes since the last snapshot, it does not take much of your internet bandwidth either.
These snapshots that only record the difference since the last are called incremental snapshots. To make sure you don't end up with too many of them, occasionally the device will do a compaction, which allows it to throw old incremental snapshots away. You can learn more about this on our storage page. But the short and sweet of it is that your data is always backed up and can be instantly restored. A nice side effect is that is gives you the ability to travel back in time, very similar to Apple's Time machine.
We don't just use this system for the worst-case situations, we actually use this to improve the reliability of your applications. If you have multiple devices, and one of them loses internet connection, perhaps because a family member decided that cable had to go, then Mosaic can instantly migrate the application to a different device. It does that simply by restoring the entire application from the backups on another device. Isn't that neat? We've built it so that it is fast, requires no manual intervention, and keeps your data safe from catastrophes, prying eyes and accidents.
Connectivity¶
The last challenge for self-hosted applications is what we call the ingress connectivity. What this means is explained rather simple: your home internet is designed for you to be able to access outside stuff, meaning that it lets you make connections to the outside. It is not really designed for the outside to access things in your home. In fact, often that is something that is explicitly protected against, because otherwise insecure devices inside your home might easily be hacked.
But then, how can other people access applications that you are running at your home, if it is so difficult? This is where we're helping you again, with technologies like WireGuard. Fractal Networks runs servers called gateways, these are small, low-power virtual machines running somewhere in the cloud. Your Mosaic device at your home uses the WireGuard protocol to establish a connection to one of the Fractal Networks gateways. This is possible because it is an outgoing connection. But then, through this connection, the gateway sends requests through to the device. This process is called tunneling.
One neat trick that we have up our sleeves is related to encryption. Traffic on the internet is typically encrypted using a protocol known as TLS, which is also used by applications on Mosaic. All of the traffic for the applications is going through our gateways — so how do you know your data is safe? The design of the Fractal Networks connectivity solution is such that any traffic to applications is fully end-to-end encrypted, the gateway simply passes the encrypted traffic through the WireGuard tunnel and it is only decrypted on your Mosaic device. All of the tech we build, we make sure that your privacy is respected.