foodogsquared/wiki
1
0
Fork 0
mirror of https://github.com/foo-dogsquared/wiki.git synced 2025-01-30 22:57:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
master
wiki/notebook/journals.learning-how-to-sysadmin.org
Gabriel Arazas 110e536c52 Add the last days of the sysdamin journal 
I forgot to commit these. I really need to clean up my notebook to make
it easy to transition to public journals (or maybe in the blog).
2023-08-23 17:13:19 +08:00

90 KiB
Raw Permalink Blame History

Journals: Learning how to sysadmin

Goals and expectations

While I've been managing a Linux desktop installation, it isn't the same as adminstering systems in a server setting. So I want to learn by properly managing a server in some cloud platform (e.g., roam:Amazon Web Services, roam:Google Cloud Platform, roam:Microsoft Azure). I would want to manage one (maybe with a roam:Debian Linux installation) in at least a quarter of a year and get to a level of getting a job out of it.

To be able to know the lay of the land, I'm able to find job listings from various sources 1. The average list of skillset required seems to lean on the following:

  • Knowledge in Linux components

    • roam:nginx
    • roam:Shell scripting

  • Development deployment tools, mostly related to containers such as…

  • Knowledge in managing Windows servers

    • Active Directory
    • LDAP user and group administration
    • roam:Powershell

This means I would have to start at 2022-11-10 and reach that level at least in 2023-02-10.

Resources, preparations, and execution

The simplest way to get started is to maintain a server for a variety of purposes. You could maintain a server for…

  • A password manager for your private accounts (e.g., Bitwarden or Vaultwarden).
  • A game server for your friends (e.g., Minecraft).
  • A code forge to store your own code (e.g., GitLab, Gitea).
  • An audio server that can be used to play anywhere on your devices (e.g., Funkwhale, Navidrome).
  • A content management system to serve your organization (e.g., Wordpress, Grav).
  • A web server for your web applications.
  • A chat server for your community (e.g., XMPP server, Matrix server).
  • A social media instance for your community (e.g., Mastodon, Pleroma, Misskey, Lemmy).

In order to get started, you have to choose a virtual private server (VPS) host. There are several of them to get started including the big 3 web services (roam:AWS, roam:GKE, and roam:Microsoft Azure), Digital Ocean, Hetzner Cloud, Linode, and vultr. Each of them has a starting tier (with the big 3 has a free tier as long as you have a valid credit card information) that is under $5 dollars in average.

Alongside the list of services that you might be interested in, there are a lot of applications and services to be interested.

  • You could look into the services offered by Yunohost, a Linux distribution that is primarily targeted to self-hosting services.
  • There's a lot of services hosted at awesome-selfhosted/awesome-selfhosted from GitHub.
  • Any services from roam:NixOS modules.

Take note, each daily entry is made progress with 4 pomodoro sessions (2 hours) a day. Then, the writing is done after, trying to recollect my experience as much detail as possible. Anything more should explicitly be listed in the entry.

Habits and todos

DONE Learn kubernetes and Google Kubernetes Engine

SCHEDULED: <2022-11-10 Thu>

DONE Deploy a Vaultwarden instance

DONE Deploy an Archivebox service

DONE Deploy an RSS feed service

DONE Deploy a cluster

TODO Manage a Linux server for at least 3 months

<2022-11-10 Thu>–<2023-02-10 Fri>

TODO Manage a Windows server for at least 3 months

<2022-12-10 Sat>–<2023-03-10 Fri>

TODO Deploy a NixOS image in virtual private server host

DONE Deploy with deploy-rs

DONE Deploy in Google Cloud Platform Compute Engine

HOLD Deploy in Microsoft Azure Linux VM

DONE Deploy in Hetzner Cloud

TODO Self-hosted services

DONE Vaultwarden

DONE Gitea

HOLD Sourcehut

DONE OpenLDAP

HOLD MinIO

DONE OpenVPN

DONE Wireguard

2022-11-10

Started to journal my journey for system adminstering up to semi-professional standards. For now, I'm scouting my options though I previously tried with Google Cloud Platform and deploying a Kubernetes cluster on it. I might manage a Linux virtual machine right away using Compute Engine from Google Cloud Platform. I'm very tempted to make it with a NixOS image as similarly laid out in this blog post however I'm going with Debian as it is closer to traditional setup. I may even consider something like Red Hat Linux or Rocky Linux.

2022-11-11

Retested the installation for cert-manager with their page with Google Kubernetes Engine and I still didn't able to successfully complete using only a raw IP address. Opportunity to buy a domain for myself and follow it the next time.

I'm very very tempted to make a NixOS deployment image as I've already seen what I can do with it but for now, I'll stick with the traditional tools. However… managing both is an option. :)

2022-11-12

For now, I've been just managing a Debian virtual machine and successfully launched a publicly-accessible web server. Mostly involves enabling the service for the web server and configuring the firewall. It cannot be accessed easily since the instance's external IP is ephemeral. As for letting HTTPS access, there is no such thing since signing certificates is only done with domains and I'm only accessing the server with a bare IP address.

This is the first time I have to worry about the lower-level things I haven't touched with the usual processes such as deploying websites and all.

Anyways in case you're curious, why Debian?

  • It is stable. Though, I have options such as deploying images with NixOS.
  • Support and community is large. It is a battle-tested distribution with a large package set and lots of resources have been written for this system.
  • Consequently, it has extensive documentation not only for beginners but also for various aspects like its releases. While its community wiki is not as thoroughly documented as Arch Wiki, it often contains enough information to get you by when managing a Debian server.

2022-11-13

I haven't done anything much in this day. On the flip side, I'm overworrying about the price considering I'm in free trial and Google will only charge if I opt-in to activate the full account. It turns out it isn't much of a worry if I leave it alone. Having an ephemeral external IP address and being so low-value might have something to do with it.

For now, I'm going to plan to create a simple wiki server with the traditional LAMP stack. What is it going to contain? Simply my findings and mainly for configuring Mediawiki as I'm assuming the role of a sysadmin.

2022-11-16

Unfortunately, progress has stalled for now since I don't have a usable bank account for now. Once it is available, progress should be quick with the availability of a domain name for me to mess around with.

A domain name is surprisingly affordable. Just the services attached to it is where most of the expenses come from such as the domain email hosting and whatnot. For the domain registrar, I picked Porkbun since it has a lot of sales and it is generally cheaper than something like Google Domains.

2022-11-22

Setup my own blog with the domain. It was slightly confusing at this is my first time diving into the level of server and domain management. First time encountered concepts like the DNS, CDNs, and managing DNS records, all of which I've learned from Cloudflare documentation of all things.

With the DNS management in place, I mostly learned how it interacts with the servers and makes it discoverable for other servers such as the effect of DNS caching which can take under a day to take effect.

The main problem I have encountered is redirecting my blog in https://foo-dogsquared.github.io into my custom domain. However, I soon found out that all of the pages under my GitHub Pages domain is affected, making all of my project pages part of the domain. The "easy" solution is just deploying my blog into a separate service and deploy the main GitHub Pages with a redirection page. It is also the time I haven't used Netlify in a long while.

The chain of problem never ends as now I would like to deploy my blog with Netlify easily. Unfortunately, Netlify doesn't have an easy way to install and bootstrap an installation of Nix package manager unlike using with GitHub Actions. A solution for this would be using GitHub Actions to build and Netlify to deploy which fortunately someone has already created a solution.

Most of the problems I have are from misunderstanding and misconceptions of how DNS and server management works. One of the prominent misconceptions I have is the DNS management is completely on the server, neglecting how clients can also affect the browsing experience. This unfortunately took me two hours to figure out and completely missing the real problem. Whoops… There are some still misunderstanding with the DNS though. I'll have to go with the basics.

I also thought GitHub Pages can be separated from domain per project page. So that's another concept I didn't easily able to wrap around my head.

Despite the fumbling around, I would say not bad. Now, I'm very very motivated to go self-hosting mode as I continue to host my personal notes (that I continue to neglect updating). I would like to self-host a Vaultwarden and Archivebox instance the next day.

2022-11-23

Another day, another time for DNS misadventures and misunderstanding. This time, most of the problems come from the misunderstanding of how hosting works which is far off from my recent idea of a hosting provider where each part of different domains can be specified to make up the frontend of your website. It turns out this is not the case.

I was able to deploy my blog with Netlify and set it to my domain. Now, foodogsquared.one is open for the world! I still haven't solved the issue of missing icons from the deployment but I'm very confident it is an DNS issue seeing as the "missing files" can be viewed, just viewed with the inappropriate headers that cause them to be blocked. Not to mention, there is missing redirections for the old site which makes it inconvenient. The only hope is nobody visits my site as I'm already dormant for the most services this year.

The thing only took about an hour where most of the time are spent in questioning and swatting the cache and tumbling over Porkbun's interface as I repeatedly reset and didn't realize my DNS records are kept being reset every time I want to point the domain to my GitHub Pages instance.

In any case, I'm just to going to delay fixing the issues from the blog site because I want to self-host some services. ;p

In this case, I want different services as part of one domain (e.g., my Vaultwarden and Archivebox instance under foodogsquared.one). It turns out that while Netlify allows some form of domain management, it simply isn't flexible enough especially since the services I put together are more likely to come from different sources. I mean, simply deploying my blog already requires Netlify for it, what more for self-hosted services that Netlify cannot simply do?

To make it possible, I have to manage a proxy server that lets me sew in those services altogether under one domain. That is, I want to access my Vaultwarden instance in vault.foodogsquared.one, I want my feeds to be accessible in feeds.foodogsquared.one, I want to self-host my code in forge.foodogsquared.one among other examples. Luckily for me, several of them exist such as Nginx, Caddy, and the good 'ol Apache HTTP server all of which listed software have capabilities beyond a simple server. However, I chose Nginx seeing as it is the popular tool in hand and also because a lot of job listings that list knowledge for nginx as part of their wishlists.

Currently, I gave in to the temptation to configure all of the servers with NixOS. Alongside the fact that I already have enough for imperatively managing them servers, there are additional three main factors to this decision:

  • The declarative configuration.
  • A framework for generating custom images (nixos-generators) that is built on top of nixpkgs.
  • The fact I already have an existing configuration that can serve as a framework to easily instantiate individual nodes.

The hardest part is creating my first image which is going to be deployed in Google Cloud Platform. The second hardest part is managing my Google Cloud Platform account as the mountainous amount of things I have to keep in mind whenever I'm staring at the dashboard of several cloud providers. The third hardest part is the amount of prerequisites before I even start doing one thing which is already a thing that the second hardest part is giving me. Unfortunate…

On the other hand, my NixOS configuration is slowly turning to be a nice monorepo for deploying everything I want. It is surprisingly easier to manage them but the part that's giving me the hardest part is the deployment. As for private files and deployments, this is easy to manage with Git worktrees which is somewhat tedious to make sure my public and private branches to sync.

2022-11-24

The configuration for Vaultwarden are in place in my first NixOS-powered deployment but most of the problems are from the lack of understanding the networking infrastructure. Fortunately for me, there is the Debian Handbook with details on each facets on the infrastructure. It is specifically aimed for Debian systems but it is good enough if you're familiar with the interface (which is just a command-line shell such as Bash).

Before that, the trouble comes from setting up a mailer which is troublesome if you only have a GMail account. However, I'm also considering to move my email provider from GMail to something else. Candidates include Fastmail, Zoho Mail, and mailbox.org, all of which has a paid plan (and also a long trial period of at least 2 weeks). In the end, I decided to not use mailing services altogether for my self-hosted services for the time being.

As for self-hosting my code, I did initially consider Sourcehut since I'm largely interested in how much resources it needs to host it. However, that didn't work out as there seems to be a lot of maintenance required for my current needs which is simple right now. I still heavily consider it for future endeavors though especially with its comprehensive documentation and integration of services is just nice to have. Not to mention, Sourcehut is still in alpha which indicates the maintainers still have plans for it. Its primary maintainer especially considers Sourcehut to be easier to self-host so the plan of self-hosting Sourcehut is not entirely thrown away.

In the end, I decided to use Gitea considering there is already a NixOS module for it (at least in version 22.11) and implementing a new way to communicate between forges with ForgeFed. This means collaboration between different instances is very much possible and I'm in support for them.

Compared to Sourcehut, Gitea is simpler to initialize which I was able to quickly start an instance. Most of the time came from viewing the configuration options and testing the instance.

2022-11-26

The deployment failed because I forgot secrets management is a thing. Each infrastructure-as-a-service apparently have their own thing such as Google Cloud Platform KMS, Microsoft Azure Key Vault, Amazon Web Services KMS, and Hashicorp Vault. It's a good thing I'm using sops for this.

It's a short time from the previous days but it should go back to normal with the time to journal this abomination. Around this time I also signed up for Microsoft Azure free tier subscription for trying to manage a Windows server this time around. It should be simpler to start since I have absolutely no idea how to provision a thing but compared to my knowledge for Linux-based systems, it is non-existent. So most of the time spent will be learning all of the concepts from absolute zero experience. Should be fun…

I've also decided to full gung-ho on deploying Linux-based systems with NixOS. I've deleted all of the non-NixOS Linux-based systems in my fleet and started generating a bunch of NixOS GCE images. Should be doubly fun…

2022-11-27

For now, I've gone back to managing a deployment with Linux-based systems and try to properly manage this time around.

For a start, I decided to manage the static websites separately from the server since Netlify apparently does not bode well to proxies. At the very least, those websites can now go at their own pace instead of deploying them altogether.

Second, most of the services are misconfigured. Classic… Most of the domains and settings are not properly configured which means I have to review the documentation for the nth time. It's not exactly a chore especially that this is my first time managing all these things. Not to mention, NixOS surely does some things differently sometimes which does not go well with me especially that I rely on resources that are mostly written with the mainstream distributions in mind (i.e., Debian, Ubuntu).

Finally, I'm now going to add one more component into my server which is PostgreSQL. All of the services I've used so far can be configured to use SQLite which makes things easier but SQLite is mainly made with the filesystem in mind unlike PostgreSQL which is primarily made with network services. Fortunately for me, its documentation is easy to follow.

At least for tomorrow, I plan to manage one more component into the mix with an LDAP server for user and group management which has a lot of presence in job listings that I've seen. Fortunately for me, there is an OpenLDAP service module already available in NixOS. I just have to be careful in chewing in managing this seemingly simple server.

2022-11-28

Welp, most of the configurations of the services should be fixed but the last thing that remains is proper deployment with the secrets. While I could do that by simply transferring the private key into the virtual machine, it just misses the point of using a key management system which GCP already has. Pretty much, I'm missing out on it if I didn't use it so I have to use it. :)

From what I can understand, with sops, you have to set the proper credentials to be able to decrypt it. That's fine for local development environment but it isn't nice for deployed systems. One of the ways to properly set it is by using a service account with the proper permissions which in this case for encrypting and decrypting GCP KMS keys.

So I created a user-managed service account to be used for the server, set the proper permissions, and make the user-managed account impersonate as the default service account because I don't want to manually manage that. Be sure to read up more on how to properly manage service accounts.

The reason why I laid it all down in this writing is because the documentation of Google Cloud Platform is surprisingly nice to use… sometimes. The way they show different ways to accomplish a task with different tools (e.g., Console, gcloud) is a nice touch. However, the amount of looping links makes it easy to get overwhelmed. Am I the only who just repeatedly visit between different pages just to get the idea from a single page? I understand the reasoning as a knowledge base that caters both to new and experienced users but it is something that I experienced.

I feel like the process of simply doing those steps previously mentioned should take way shorter time than it should be. Most of the time is spent in staring at those pages, trying to see if I'm following it right. This is where I feel like I should've really started with Qwiklabs which I didn't is a thing when I started. Welp…

2022-12-01

Here we go, start of December. Only two months to go before the deadline to become halfway to professional-level (or at least getting paid).

I haven't done anything from the last two days so there's no entry for it.

Still having some problems, mainly from PostgreSQL service this time. I'll use this opportunity to experiment debugging and maintaining services with PostgreSQL. Thankfully, its documentation is very comprehensive especially that it has a dedicated chapter for server management. I'm only starting to wrap around my head with the concepts of a database and its management.

The errors from the database service are more likely from the lack of proper privileges. From the Vaultwarden service, the new error this time looks like this.

Dec 01 01:41:03 localhost vaultwarden[762]: [2022-12-01 01:41:03.533][vaultwarden::util][WARN] Can't connect to database, retrying: DieselMig.
Dec 01 01:41:03 localhost vaultwarden[762]: [CAUSE] QueryError(
Dec 01 01:41:03 localhost vaultwarden[762]:     DatabaseError(
Dec 01 01:41:03 localhost vaultwarden[762]:         __Unknown,
Dec 01 01:41:03 localhost vaultwarden[762]:         "permission denied for schema public",
Dec 01 01:41:03 localhost vaultwarden[762]:     ),
Dec 01 01:41:03 localhost vaultwarden[762]: )

The error is a bit intuitive with the intuition being a permission error with the 'public' schema. I've simply resolved the error by adding the permissions from the NixOS config like the following snippet.

{
  services.postgresql = {
    enable = true;
    ensureDatabases = [ vaultwardenDbName ];
    ensureUsers = [
      {
        name = vaultwardenUserName;
        ensurePermissions = {
          "DATABASE ${vaultwardenUserName}" = "ALL PRIVILEGES";
          "ALL TABLES IN SCHEMA public" = "ALL PRIVILEGES";
        };
      }
    ];
  };
}

As an additional fact, I've quickly come across from the documentation that 'public' schema is the fallback schema for databases without names. That's something to keep in mind in the future.

But anyways, here's a light writing on the summarized version of my understanding of the database starting with its authorization process, the part where I'm spending the most time on understanding it.

Being accessible through different ways, widely available to other users, and globally contains various application data, the PostgreSQL service has ways to make sure access to the database is only done by trusted users. This is where authorization comes in similar to POSIX-based systems when authorizing access to various services.

Inside of the database, various services (which serves as clients) want to access their data which the database contains a variety of them. In order to safely access them without much problems, PostgreSQL plants some ways to verify its client. There are different ways PostgreSQL can authorize access to different users.

  • Tried-and-true password authentication for the user it tries to access as.
  • LDAP authentication.
  • Another way is simply leaving a map of connections and their trusted users in a file (e.g., pg_hba.conf).
  • Accessing the database as one of the user of the system with the same name as the user of the database. This makes sense: it is more likely a dedicated user specifically created for a certain service alongside a database for that service. This authorization is referred to as an ident authorization. Several examples include hosted services with dedicated setup (e.g., user and user group, database) as they're logically mapped from the operating system and its different components.

As for the plan to maintain an LDAP server and user management with it, I'll start around this week hopefully. For now, the focus is debugging and maintaining a server. Mainly, by SSHing into the server and getting used to the maintenance tools with systemd.

There are also some things I've learned such as:

  • Creating a new unit file easily with systemctl edit --full --force $UNIT and it will just place it in one of the unit paths.
  • Easily viewing how much journal files took space with journalctl --disk-usage which also supports it at systemd at user-level with --user flag.
  • Flushing all ephemeral journal files from /run to a persistent storage with journalctl --flush.
  • Log rotation with journalctl --rotate.
  • Ports lower than 1024 is a privileged port and normal users cannot use it. 2

I'll get around to using a traditional Linux distro (Debian, again). While server management with NixOS is nice, I think getting used to the traditional environment nets more credibility. Though, it is getting easier to map concepts I'm getting from NixOS to the traditional setup with time. Especially that most of the things from NixOS are setting up services which could be done in any Linux environment anyways.

While I'm at it, I'm starting to look into backup services. I'm already using Borgbase with only the free 10 GB but it quickly ran out. For now, I'm looking for a cheaper option if there's any.

2022-12-02

Hey, the Sendgrid account application has been approved. Well, that's one more service component to enable.

There's also one very stupid mistake I didn't realize I'm making at this point from the very beginning: I keep hitting the duplicate certification limit and I keep forgetting to backup the certificates. I KEEP DOING THIS FOR THE PRODUCTION DEPLOYMENT, WHAT THE HELL! I always thought the production only makes the same error as the development deployment. Welp, I'll just wait for a week then everything should be fine and dandy for the production. I'll just use test environments with domains with test subdomains on them while I'm at it. Or I could just generate a new request with a different "exact set" of domains as already mentioned from the Workarounds section from the aforementioned page.

2022-12-03

Today's theme for misconfigurations are permissions. Permission error for inaccessibility for PostgreSQL schemas for a certain user, permission error because the users does not have the permission to access the files, et cetera, et cetera. Though, we're now in the home stretch without much shenanigans. I'm now slowly chewing all of the things I bit off.

On the other hand, I found out about how certificates are generated in NixOS with the default workflow. It is using lego which has support for tons of DNS providers including Porkbun which is the domain registrar I used to buy my domain name. Nice! Now, there is more automation including taking care of setting the appropriate DNS records and now the shenanigans with certificates is on the past (or at least significantly reduced). All I have to take care of is my secrets file and managing my servers with some security which I created the following initial guidelines as a start.

  • Starting with removing the keys from various KMS such as from Google Cloud Platform. This makes it easier to decrypt keys if someone has access to one of the users from its respective virtual machine instances since all of them are configured with their respective KMS enabled. So they have to go.
  • Giving some more thought about managing secrets with their respective keys. Some examples include giving the least privileges with the minimal number of keys plus a fallback key for emergency. 3

Also, while taking care of that, I also found out about Porkbun API which means I could create a simple program to interact with my DNS records instead of going to the website. There are still some things to set up in the website but that's not much of a problem.

2022-12-04

Took a more serious approach to learning PostgreSQL this time because of primarily two reasons:

  • Apparently, there are changes related to the public schema which is where most of the problems are from. That is indeed something to keep in mind as /foodogsquared/wiki/src/branch/master/notebook/2022-12-01 has already mentioned.
  • I only haphazardly glanced of the documentation before. Really, the approach is more scattered: just debug the errors from the services and go to the appropriate chapters. Which is fine for the most part but it can completely screw me over if I'm not mindful of the concepts.

With the listed information in mind, I deduced the problems are really coming from the applications to use the default schema which is in public (which is problematic for my configuration). But I missed a very very crucial information that made me bash my head to this simple problem for hours over the previous days. Apparently, most of them services can be configured to use another schema. Yeah. That's it. What the hail…

However, there are consequences that made me understanding a bit how to properly manage a PostgreSQL database.

Specifically, what I've done after is trying to understand is how permissions and other objects are handled through the entire clusters. More specifically: schemas and roles which my understanding is proven to be lackluster.

With the new understanding, I came across a new way to manage them schemas. The maintainers of PostgreSQL encourages a more secure usage pattern for managing schema referring to as secure schema usage pattern. In the encouraged practice, database users have their own schema (with the similar name for their schema, of course) and encouraged to set the schema search path to be set only for the user themselves (e.g., search_path = "$user"). Then, with the applications lending a setting to configure the schema, you could make use of the appropriate schema for the appropriate user for that certain service.

I can see what the maintainers are encouraging here. With the different ways to authenticate (especially with the ident authentication and the like), each service which may be running with their corresponding operating system user has a corresponding database user that owns a corresponding schema for the service data and whatever objects they want to access. It's a nice, easy-to-understand practice that easily maps between the operating system and the database objects. With the secure schema usage pattern, you're essentially playing Match-3 for the sake of simplifying the system. Anything that reduces cognitive load for that is good enough for me.

On the other hand, the workflow for deployment NixOS-based systems is great with deploy-rs. Though, the only issue is I cannot deploy the config with user passwords. The terminal is usually mangled up while using it with the SSH'd process. Thankfully, one of the comment has shown a solution by disabling magic rollbacks which is unfortunate as I really would like to have those but it works. There is another comment that shows what's really happening under the hood which I can confirm with the part about how the input is echoed and fits with my experience of seemingly janky input with the mangled output. Another quick solution is configuring the SSH user to be passwordless with sudo which is not great to me. Don't like passwordless sudo especially that I've set a password for the user, making it essentially useless.

2022-12-05 - 2022-12-07

Nothing much happened here aside from sporadic debugging sessions. I took this opportunity to learn how to debug the system mainly with systemd and PostgreSQL components.

More specifically, some new things found:

  • I found how great systemctl show subcommand is. You can view the properties of a unit which is nicer for debugging systemd services. Not to mention, you can specify the properties with the -P flag (e.g., -P User, -P Group).

  • Some facts with networking ports. This is not something I managed over the days since Google Compute Engine firewall has an easy setup.

    • Ports under 1024 are considered privileged (or system) ports and normal users cannot use them. Ports on 1024-49151 are registered (or user) ports and are maintained by IANA to assign and map ports to services. Ports on 49152-65535 are private ports and often ephemeral. 4
    • There are conventional port numbers used for certain services especially in privileged ports.
    • IANA does have a registry of port numbers for services with databases in different formats available for download.

2022-12-08

TL;DR - I've looked into managing emails with my system which is a responsibility for administering systems, yeah?

Anyways, I've been looking for an alternative email hosting provider that is not Google and found out about mailbox.org from Privacy Guides. They happened to have an annual Christmas campaign which nets me free usage of their standard plan for six months. Talk about timing.

What makes mailbox.org odd to me is the amount of services they provide and its pricing. For €3 (about ₱180), you have basically the equivalent of Google Workspace suite including 10 GB of space for email (which is huge for me excluding mailing list discussions) and 5 GB of cloud storage which is not much but it is nice. I would overlook mailbox.org's other services and only in it for the email provider buuuut I have found some neat things about their services. I first misjudged that it would be just some low-quality office suite services with the email service as the forerunner but it apparently uses Open-Xchange (OX) which means it can work with OX-compatible apps. Not to mention, their interface and infrastructure are built on it which the maintainers of the service are also involved with the development of the OX app suite. Aaaaand that opened up the world (or rabbithole) of office suites to me, out of all things.

Aside from the services, there are some neat things with the email service itself. One such example is secure emails which is sent through an email address through secure.mailbox.org. 5

Sorry for misjudging mailbox.org as it turning out to be a solid provider the more I look into it. It turns out what mailbox.org provides is an office suite, basically an all-around alternative to Google Workspace. I can see myself being their customer for years though I'm mostly feel like it's a waste especially I'm looking only into email hosting. However, it is nice for a non-Google Workspace/Microsoft 365 setup if you want a more privacy-oriented workspace suite for your family or group.

At any rate, there are some services that are exclusively focused on email hosting such as…

  • MXroute which has a nice option considering you have unlimited domains with unlimited email addresses. It costs $45 for a year which is cheaper compared to other services but it isn't for me (and I just missed the Black Friday sale, CRAP!). Comparing this option to my current needs and situation (only using one domain and using mailbox.org), it isn't the suitable choice considering the price is slightly close (without sales discount or anything).
  • Purelymail is another service that is purely focused on email. As it motto says, "Cheap, no-nonsense email". It only costs $10 a year and already allows custom domain

One criteria I left out in the list is the spam filtering. Ideally, it should be done on the service provider side which is present on all listed alternatives so far. The question is how active and the quality of the spam filtering service. MXroute seems to be more on the active side and reportedly has more positive review especially with the way how emails are sent through their service. As for Purelymail, I don't see much things in that side.

At this point, MXroute is going to be my secondary choice for email hosting. For now, mailbox.org is enough for my needs. Both services required to be configured for sensible spam filtering anyways and I am currently under free credit anyways so I should have time to know about some details about email service in general.

There is also little trivia I've learned specifically with email aliases. In mailbox.org, there are multiple email aliases that can be acquired for free (e.g., postmaster@ADDRESS, hostmaster@ADDRESS, webmaster@ADDRESS, abuse@ADDRESS). This is apparently a convention for it, specifically RFC2142 that highly encourages certain aliases for different parts of the services provider. Here's some examples from the document:

  • abuse@ is for public inappropriate behavior.
  • webmaster@ is for handling HTTP but practically it's the account for handling issues in the website.
  • postmaster@ is for SMTP, typically used for mail services.

2022-12-10

The server is mostly operational but it required some non-declarative setup beforehand. Not exactly close to what I'm aiming for but it is close enough. Really, the problems mostly comes from the way PostgreSQL 15 handles the schema which I would like to take advantage not only with the latest improvements on the package but also its recommended practices. It seems like the current service NixOS module is not modelled after those practices. The practices I tried to apply are highly encouraged in version 15 especially with its changes. The NixOS module tries to cater for the majority of available versions.

On the other hand, I've configured my server to have backups for anything appropriate though most of the focus is on application data. Most of the services have a documentation for dumping data like in Vaultwarden, Gitea, and PostgreSQL. All I have to do at this point is to back them all up with Borg which I have a remote backup hosting at Borgbase.

It's seriously NICE to have the configuration coming together in a neat little package. Aaaaaaand the LDAP server is becoming more of an afterthought. Aaaaaaand speaking of afterthought, I'm considering to host a VPN service for my own connections.

On yet another hand, I've been considering about the VPS hosting provider, preparing to move away from Google Cloud Platform. It is a shame as the platform is a nice tool. I easily created an automated workflow that'll deploy my existing configurations in the cloud somewhere especially with gcloud tool. I've considered Hetzner Cloud as the VPS host lately as the server options is ridiculously cheap. About €5 for a complete server with a public external IP compared to about $30 in Google Cloud. It also has a command-line utility humorously named hcloud which I should be able to easily automate it.

2022-12-12

The test Linux server is now mostly operational. It is nigh time for managing a Windows server and make them communicate with each other, probably with services with Active Directory which is apparently an LDAP server with Windows-specific bells and whistles.

Today, I've also learned about the existence of systemd-tmpfiles for customizing Gitea from my NixOS configuration.

However, there are some things left to do in this production server in development. Most of them are involved with authentication services.

  • For one, I would surely create an LDAP server just for the kicks, specficially kick up an OpenLDAP server.
  • I've decided to add one more authentication service, mainly for the web with single sign-on and social logins. This is apparently a separate service to an LDAP directory service. For this, I decided to host with Keycloak.
  • Add the appropriate settings for the already existing services. Since the additional services listed here require more caution for using it, we may as well make it for the rest of the system. This includes adding secure TCP/IP connections with SSL for my PostreSQL service.

Aside from these authentication services, I reviewed my understanding of CA certificates in relation to HTTP(S). As HTTP is stateless, it isn't concerned who is communicating between each endpoint. However with HTTPS, it is a different story.

HTTPS is essentially HTTP + SSL/TLS. 6 From what I can understand in TLS, the certificates are composed of a keypair: one issued as a public key for clients to communicate with the server and the private key to verify them so the server can communicate with the client securely.

In order for applications to make use of this, you can either configure them to point the certificate files. This is commonly used for web servers (e.g., Nginx, Apache, Caddy). There are also other applications that make use of this such as databases (e.g., PostgreSQL, MySQL), Kubernetes, and authorization services (e.g., Keycloak).

2022-12-15

Today's theme for management is: secrets management. While this is already done for my NixOS setup which is done with sops, keeping those secret keys is now a matter of securely keeping it.

To solve this problem, we have to lay out all of the information of our current situation:

  • There are private keys for different formats: GPG, SSH, and age. Not to mention, remote secrets such as from GCP KMS, Azure Vault, Hashicorp Vault, and AWS KMS.
  • Proper storage for these keys. This is especially important for GPG where it revolves around your identity. As I don't have an iota how to do it right, I followed someone's guide for this instead. More specifically, I followed the recommended resource from that post which is from the subkeys management page from the Debian Wiki.
  • Multiple keys management. I want to properly learn how to manage them keys for different purposes.
  • Backing up properly which is already done with borg. Hoorah for me…

2022-12-18

Now, the start of properly setting up them authorization services. More specifically: Keycloak and OpenLDAP.

  • For OpenLDAP, the project has a nice documentation for configuring and administrating an LDAP server. I practically have no choice since other alternatives such as FreeIPA and Authelia is not exactly present as part of the available NixOS modules.
  • For Keycloak, the documentation has a nice structure to it including references and "Getting started" guides.

Though, all of the time are spent on learning OpenLDAP instead. Looking at the examples, it is quite verbose. It is becoming similar to the Google Cloud Platform documentation problem where it is becoming overwhelming and requires through pages back and forth. Don't get me wrong, it is nicely structured but it is verbose. I think I need some more time to absorb this. Even if I skimmed it, I cannot get a bigger picture clearly.

Looking at the whole picture, it is pretty simple as to what it is. Most of the details are hidden behind conventions which is where my problems lie. It really requires familiarity which is acquired with time. Not to mention there's not much examples I can see in the whole wide web so I cannot freely experiment some things. (Thank good God for NixOS enabling easy experiment with building VMs easily though.)

I'll put up with studying about Keycloak later. While I can see myself using Keycloak more, LDAP seems to be more common judging from the job listings which is why I'm studying this in the first place. Active Directory being one of the most prominent example for this. Speaking of which, I should really get with managing a Windows server at some point. Preferably, they would have to be done in early January as I'm not done configuring the (NixOS) Linux server yet.

2023-01-01

Took some holiday vacation. Back to normal scheduling.

I've been deciding on migrating host to Hetzner Cloud since it is way cheaper compared to Google Cloud Platform. The disadvantage is there's no way to make into provisioning with a custom ISO except with a kexec-based image. However, you can easily initialize a NixOS system with nixos-infect, a script that converts a traditional Linux system into NixOS.

That's pretty much it for the most part. Most of the things done is only for migrating and getting familiar with Hetzner Cloud. It is pretty light on the thing but most of the system are not working due to ACME. I'll just wait for some time and tinker with it some more.

2023-01-04

I'm now trying to get a concrete step into showing some credentials by completing certain courses in Coursera. It is mostly for certificates just to show that I have self-drive. Fortunately, Coursera has a financial aid initiative which is quite easy to get approved of but it does take some time.

For now, I've been completing a part of the Google IT Support Professional Certificate, mostly in the security side as it is the weakest side that I know. I'm also taking the Cloud-Native Development with OpenShift and Kubernetes specialization also for the certificate but learning more about Red Hat systems seems to be interesting. Fun and beneficial: double the benefits!

For now, the self-managed server has been stopped since I'm just waiting for the certificate cooldown. The server should be backed up a couple days from now.

2023-01-06

The server is back up and running!

I also added an Atuin sync server especially that I use more often now and slowly needing to sync the history. While the sync server from the developer is nice, I decided at the end that I would like to manage it for a try. Fortunately, the service is already available as a NixOS module so there's not much problem there.

Next up, I also replaced the OpenLDAP server to be managed with Portunus which is already available as a NixOS module. I haven't tried completely managing it yet, though. I only scratched the very surface of managing and using LDAP. I should find out more about it once I continued with my courses I previously mentioned in this journal.

There is another thing which is adding a new component in my server: a nice storage box. Specifically, I added a storage box from Hetzner which is surprisingly cheap with €4 for a terabyte. What's more, this storage box has some neat features including an SSH and Borg server. This instantly replaces my decision to go with a Hetzner storage box instead of Borgbase which is nice but with the budget in mind, I get more mileage from Hetzner's offer.

A complete self-managed setup for €8 a month? That is very nice especially with a budget. I'm very satisfied with Hetzner so far.

Even with a meager job pay, my whole setup is quite cheap. Here's a table of the expenses.

Thing Expenses (in EUR per month)
Hetzner VPS 5.35
Hetzner storage box 3.5
mailbox.org account 3
foodogsquared.one domain 2
Total 13.85

2023-01-11

Looked into properly configuring fail2ban which apparently does not do much by default.

The documentation of the project is a bit scattered throughout its website, wiki, and the source code. Most of the knowledge I picked up came from the already existing configurations from upstream with the manual as the accompanying starting point all to make things connect.

Another thing that is neat is it can handle systemd journals apparently. This make it easier for me as I would like to keep my services managed with systemd.

At this time, this is where I learned about matching with journal fields which journalctl is primarily used for. Even the -u UNIT option that I always use is just generating certain journal fields under the hood. My usage with journalctl is pretty basic as seen from Command line: journalctl. I only done like the basic matching of a unit, some basic journal management (e.g., log rotation, pruning), and monitoring them services.

Overall, diving into configuring fail2ban is not exactly a great experience. Reminds me of the situation for the Nix ecosystem: it is a great tool hindered by its sorry state of documentation. Except this time, it's somewhat worse with the outdated manual and the scattered state of picking up the pieces together. I don't know enough to make more insightful comments but this is coming from my experience as an outsider trying to dive into using it. Delving into Nix has prepared me for this type of situation and I'm not liking it.

At the end of the day, fail2ban is a great tool just hindered by its documentation. The lack of good user documentation just means you'll have a harder time getting to know things if starting out and wanting to know more beyond the basic things.

To quickly get up to speed with fail2ban, I recommend starting out with the previously linked manual and see the upstream config files for examples which is commented at some parts. Some of the comments from the upstream config files are even repeated with basic documentation as if someone is expecting an admin to just jump in to see how things work behind-the-scenes which I think it is what is intended.

2023-01-12

Just a quick update on fail2ban regarding its state of documentation: apparently it has manual pages which I completely missed because the package from nixpkgs doesn't have them. I added it to the package definition and created a pull request for it.

Now, the manual pages are the user documentation that I'm looking for. It's actually very nice complete with details starting with jail.conf.5 being a go-to reference and its manual pages for executables (e.g., fail2ban-regex.1, fail2ban.1) is nice and brief. Sure, it's scattered but that's just the state of being a Unix manual page. I'm retracting my statements which I unfairly described the state of documentation as a poor one. Instead, I'm replacing it with it's pretty good and the opinion of Nix is a great tool with poor documentation is still unchanged. :)

On the other hand, I started configuring with Portunus as an LDAP server. I really want to make this work despite having not much use out of it just for the sake of learning other ways how to authenticate outside of the web logins. One of the handy features with Portunus is applying a seed file which essentially declares the groups and users for that LDAP server.

I've also started to modularize my Plover NixOS config since it is getting started to become really big. An embarassing side story with this is that I once accidentally deleted the modularized Nix files and I have to rewrite them, not realizing that Neovim still have those files as a buffer stored in-memory. So there's one thing that is hopefully helpful to remember next time you find yourself in that kind of situation. Most text editors can do this including Visual Studio Code and Emacs so you can just restore them back by saving them. Though, if you close those buffers in the event of deletion, it's deleted for good.

2023-01-13

Continued my progress of the self-studying for the Coursera certificates. Despite seemingly only doing it for the certificates (which you could describe it like that and I'm not denying any of it), the courses I've chosen is pretty interesting. Especially with the IT fundamentals specialization from Google. I mostly proceeded to the part where they discuss about cryptography and its applications.

There is an interesting thing going on especially that it is leading me to several stuff such as the practical uses of a VPN. I always thought a VPN is mainly used to anonymously browse the internet but it turns out, it isn't the case. A VPN service can be used for several things.

  • Getting around geo-restricted content.
  • Safely accessing sensitive services from a different network (e.g., your VPS's, your own home network) from an outside source.
  • Piracy.

You can still use a self-managed VPN in a VPS as a privacy tool but that will depend on your trust towards the hosting provider and your care towards privacy (if at all). However, you're pretty much limited with the configuration of the system such as the location which is most likely only hosted on one location which is pretty much the main incentive for using a VPN service provider such as Mullvad and NordVPN. Not to mention that you share the self-managed VPN service with the VPS provider. It is a matter of judgement whether you tolerate that.

So far, I've only considered self-managing a VPN service in my VPS instance which is most likely happening anyways. For now, I'm just scouting for resources for configuring OpenVPN which is already available as a NixOS module. I have a feeling I'll have some difficulty facing in the next day. Especially with networking parts of the system configuration.

2023-01-14

My feeling of dread from yesterday is right on the mark. (I guess it's not exactly surprising that it will happen considering I have practically absolute zero experience and knowledge regarding networking.)

I started from the bottom-up knowing a little bit on networking. More specifically, I'm trying to configure my networking setup on my server starting with supporting IPv6 which is already a thing in Hetzner Cloud. I just have to figure out how to do that with my system tinkering with these network-related thingamajigs. Thankfully, Hetzner does have a dedicated page on the basics of IPv4 and IPv6 which is very helpful in making me understand the basics.

Most of the resources for configuring network-related settings for Hetzner Cloud servers are aimed for the traditional Linux distros. Searching through the NixOS Discourse instance, however, netted some discussions on that exact topic which is convenient. More specifically, this post which describes setting the network with systemd-network made me learn about setting up networks with systemd directly.

I've learnt more about IPs with the following resources.

2023-01-15

More studying about networking. Specifically, trying to configure my networking setup with systemd-networkd replacing the traditional script-based networking that is done in NixOS. I've been tempted to learn it seeing as it has some nice features compared to the traditional networking done by NixOS.

The first try is a disaster because I didn't know I misconfigured the routes for the network. More specifically, I misunderstood the gateway address is the private IP address connected to my Hetzner server, not realizing the gateway address is in private address 172.16.0.1. This resulted in the following error logs.

Jan 15 05:47:19 nixos systemd-networkd[18933]: ens3: Could not set route: Nexthop has invalid gateway. Invalid argument
Jan 15 05:47:19 nixos systemd-networkd[18933]: ens3: Failed

I'm also trying to start an OpenVPN server for the local network at the deployed server. This is also for practical purposes, hiding some of the more sensitive services such as my Keycloak instance and LDAP server. 7 Apparently, it doesn't have username/password authentication and has to be implemented by installing a module. I'll try that route but I'm also very tempted to try using LDAP for it considering it also has support for it and a valid excuse now for using it.

The OpenVPN documentation is very nice with a directory of community resources of which has things like a a HOW-TO document, a reference manual, a document for hardening OpenVPN security, and a tutorial for ethernet bridging. Not to mention, OpenVPN has a community wiki containing lots of explanation for concepts. Most importantly, it has a set of offline documents which should be included with the package when installing them. I recommend to start with the offline documents especially if you're unsure where to start with the online documents.

The only thing I fear with a VPN service is the amount of bandwidth it will send out but seeing as Hetzner Cloud has 20TB worth of outgoing traffic for free which is more than enough (an understatement), I don't know it will be a problem. With that said, I haven't fully configured OpenVPN yet as I'm just exploring the documentation and creating a basic configuration out of it. It's not yet complete with configuring profiles for the client and server but it is getting there.

2023-01-16

I've been recommended to look into Wireguard which is supposed to be more performant than OpenVPN and it is baked into the Linux kernel. Aaaaaand it is easier to use. Not to mention, it is also supported by systemd. Looking into the ecosystem, I see that Wireguard has an Android app which is very nice to my checklist. However for the time being, I'm continuing with OpenVPN especially that it has more authentication options whereas with Wireguard being certificate-based.

From what I can see, Wireguard is simpler and faster. In my understanding, Wireguard is indeed simpler in the way of configuring. In Wireguard, there is no server and it instead interfaces with peers where each peer in the network is configured with each other's public key. From the initial reading, I was able to easily configure Wireguard. However, I'm more interested in configuring it with systemd-networkd which did took some time since I'm still grokking the service. In the short run, I was able to figure out to configure a peer but eventually found out I also have to configure it to my desktop. For now, I'll leave the progress here as I'm continuing to configure OpenVPN just for the time being.

A VPN is an interesting (and a must-have at this point) component to add to my server for not only learning networking concepts but also practical reasons. Very nice. I just have to figure out how to configure it like in the following plan.

/foodogsquared/wiki/media/branch/master/assets/visual-aids/self-managed-server-plan.png

As for configuring OpenVPN, I've not yet to run a server since I'm still figuring out network devices in general. The concepts are still bouncing in my head as I'm trying to make sense where things to be put together. systemd-networkd, while interesting, is a bit overwhelming with the concepts. For this, I'll leave a short list of guidelines how to get started with it.

  • Start with the systemd-networkd.service(8) manual page. It is the root of the concepts not to mention it quickly introduces to the related components of the service.
  • Its manual pages to its related configuration files (i.e., systemd.link(5), systemd.netdev(5), systemd.network(5)) are all must-haves to read. Especially that it contains a comprehensive list of examples showcasing different setups on the later parts of the manual page.
  • networkctl is going to be your best friend for managing networking setup with system. Don't forget to refer to networkctl(1) manual page for more details.

I think I bit more than I chew, juggling between configuring the networking setup to my setup, OpenVPN, and Wireguard. It is an interesting experience as I'm trying to map things together just from reading its documentation. Though, I feel it is somewhat wasted effort as most of the time, I'm only trying to map things together while being overwhelmed. Not exactly a good practice.

I'm slowly gaining confidence with my networking concepts. At some point, I'm going to review them with a related course from Coursera for it.

Next up, I think I will configure systemd-resolved as I'm having a little trouble with understanding related parts of network configuration such as DHCP and DNSSEC. What are all these, man?!

2023-01-17

I am familiar enough with IP addresses so I'm moving on to understanding DNS. The primary reason: systemd has a resolver service so I'm interested on interacting with it.

But first, I have to understand what DNS even is. Fortunately for me, there is a comic series that explains that very same thing I'm trying to understand. 8 I highly recommend it.

At this point, I'm juggling between configuring Wireguard for the network and preparing to configure systemd-resolved at a later time (or at least trying to understand it). I'm also adding properly configuring a software firewall like nftables although it is blocked from being problematic in the associated NixOS module. It doesn't seem to properly generate a configuration without errors. Previously, I've used iptables for a short while until it was apparently deprecated so I hold it off. Though, it also had problems with the resulting firewall making the network reject all incoming connections.

So, for this firewall problem, I tried the simplest solution: trying with the most sensible and minimal configuration. I tried the following nftables-based firewall configuration with following snippet on my desktop:

{
  networking = {
    nftables.enable = true;
    firewall = {
      enable = true;
      allowedUDPPorts = [ wireguardPort ];
    };
  };
}

This works on the desktop workstation host so far. It does serve as a nice starting point for learning about nftables. There are other resources I recommend:

  • Start with the nft.8 manual page. It is the canonical user reference with nice introductions to the related concepts, syntax, and options.
  • For potentially looking out to other documents, their community wiki has a nice list of nftables-related documents.

For now, I haven't created a nftables script yet. Looking at the documents, it should be take an afternoon to learn just enough to be dangerous but make stupid decisions.

So far, my experience with software firewalls are not great but that won't deter me from it. I want to have an operating system with such features especially integrating with tools like fail2ban where it can use the firewall to completely ban the host.

2023-01-18

Welp, today's theme is unfortunate server update timing. Let's start with the end state of the server for the unfortunate time: its network became unreachable from the outside.

This story starts with an impatient person as they try to upgrade repeatedly without success similarly encountering problems as described from this issue. I cannot exactly reproduce this bug as I don't have enough understanding how deploy-rs really works but I mostly think this is a server issue. To be more specific, what really happened is I cannot successfully deploy the updates as they always end with a timeout for whatever reason. As described from the linked, this is specifically tied to the magic rollback feature as seen from the following logs from a deploy attempt:

⭐ ℹ️ [activate] [INFO] Magic rollback is enabled, setting up confirmation hook...
👀 ℹ️ [wait] [INFO] Found canary file, done waiting!
🚀 ℹ️ [deploy] [INFO] Success activating, attempting to confirm activation
⭐ ℹ️ [activate] [INFO] Waiting for confirmation event...

Anyways, as this impatient person grew tired, they decided to go with the updates but without the rollback feature. It's a fatal mistake. This is pretty much where I feel NixOS configuration rollback capabilities would be very useful.

The temporary outage is caused by improper routing configuration as I haphazardly copy-pasted the configuration from the internet without taking a closer look. The following code listing is the erroneous part of the configuration.

{
  systemd.network.networks."20-wan" = {
    routes = [
      # Configuring the route with the gateway addresses for this network.
      { routeConfig.Gateway = "fe80::1"; }
      { routeConfig.Destination = privateNetworkGatewayIP; }
      { routeConfig = { Gateway = privateNetworkGatewayIP; GatewayOnLink = true; }; }

      # Private addresses.
      { routeConfig = { Destination = "172.16.0.0/12"; Type = "unreachable"; }; }
      { routeConfig = { Destination = "192.168.0.0/16"; Type = "unreachable"; }; }
      { routeConfig = { Destination = "10.0.0.0/8"; Type = "unreachable"; }; }
      { routeConfig = { Destination = "fc00::/7"; Type = "unreachable"; }; }
    ];
  };
}

This pretty much makes it unreachable from the outside. Thankfully, it is successfully configured to reach global networks from the inside. While access through SSH is no longer possible, Hetzner's cloud console saves the day. It works by booting the server as if you're physically there so it can still be recovered.

2023-01-19

Welp, decided to take a different approach to journalling. Here's what I want to do and whether or not I completed it on time.

  • Configure systemd-networkd to properly configure network devices with automatic IPv6.
  • Learn nftables for IP forwarding and additional Wireguard setup

I was not able to learn about nftables at time. Most of the time, I've been trying to parse what is going on exactly with IP addresses and how to setup a networking setup with a system. I found myself overwhelmed with the concepts of dynamically generating IPs which apparently has multiple ways to generate them which is especially true with IPv6 which is already something that I'm barely familiar with.

For a short recap, similarly to IPv4, IPv6 have assigned address ranges for private networks. These interfaces are not going to generate and assign IP themselves, you still have to assign it. However, you're dealing with IPv6. Manually assigning IPv6 IPs is not often worth especially with subnetting. And so there are ways to generate them…

  • Stateless Address Autoconfiguration (SLAAC) generates IP addresses by its nearby network links that are sending "Router Advertisement". No additional servers and manual configuration of hosts necessary. One neat thing with this is the process is decentralized, the minimal amount of changes to make, and only requiring local information. I recommend to stick to this.
  • Good ol' DHCP server except for IPv6, referred to as DHCPv6. Simiarly, a request will be sent to the server and receives an address which can be used to automatically assign to a network link. While this can be nice for restricting what addresses can be published for that zone, it is centralized. Once the server has shut down, the network will likely fall as the IPs the links hold are dynamically configured.

One more thing: apparently, you can combine static and dynamically configured IP addresses. I always thought it is only given a choice of static and dynamic IP configuration considering most interfaces I've interacted presents it this way (or at least my impressions of it). It makes sense once you know that multiple addresses can be assigned to an interface and dynamically generated IPs have virtually no difference to distinguish themselves from statically assigned IPs. It's just IPs on the way down.

I also discovered a valuable tool which I should've thought of at the beginning of my struggle understanding IPs: an IP calculator. More specifically, ipcalc because it is in the same environment you'll be configuring them networking setups anyway.

nix run nixpkgs#ipcalc -- 2001:5eca:de53::3
Full Address:	2001:5eca:de53:0000:0000:0000:0000:0003
Address:	2001:5eca:de53::3
Address space:	Global Unicast

2023-01-21

Stumbling into IP problems. Again.

This time, it's about application services. It challenges my understanding of the relations of it with applications because I didn't know they can be hosted in a different interface other than localhost. Now, those host options or what have you makes sense. The solution is to simply reconfigure them to go to a different host interface. The most tedious part is manually assigning and remembering them so I put the interface hosts in a set and just refer to that instead.

With this in mind, I fell into a mini-rabbithole for networking-related things. Most notably, I was looking for a way to automatically assign IPs to applications if possible.

The closest thing I have seen so far is network namespacing which is a thing in a Linux kernel. Aside from isolation and controlled sharing, network namespaces allows you to assign prefixes to interfaces. This seems to be fitted for my use case for a way to assign IPs to different services without manually assigning them. I checked to see if systemd can do this which apparently isn't. However, there is an interest for it and while there is an impending implementation for it, it seems to be dormant which is unfortunate considering fellow systemd contributors also expressed interest for this feature to manifest.

Another point of interest I was in is "properly" deploying a Keycloak instance. I haven't managed it since I was supposed to after I configured the VPN or whatever tunneling service I want to manage. This is where I found an alternative to Keycloak named Zitadel. It seems nice considering it can be self-hosted and deployed from a single binary. As of this entry, there is no package, module, or even just a mention from the nixpkgs repository. Seems like a nice time to try out packaging and creating a module out of it. But right now, I have no interest in fully self-managing it considering Keycloak is a popular option.

While I found an alternative for Keycloak, I also found a complement for Keycloak (or at least the type of service that Keycloak offers) called privacyIDEA which focuses on 2-factor authentication. While Keycloak supports 2-factor authentication, it is only through TOTP/HOTP. privacyIDEA supports more than that through its ecosystem of modules. Not to mention, privacyIDEA has a Keycloak provider, making it nicer to integrate between the two. I may consider to add it in my half-full plate of self-managed services.

I'm also reconsidering to deploy back to a bare OpenLDAP server but it may be just me. Portunus is pretty great so far but I didn't properly made (nor even connect to) a LDAP profile yet. I need to properly configure Portunus especially that it has options to stay only in the private network. It's just not an option with its NixOS module which should be trivial to add. I'll have to keep in mind with a PR in the future once I properly deployed the previously mentioned services.

2023-01-22

As for properly configuring my network interfaces, they are assigned with a DHCPv4 server which apparently can take some options which you can do in systemd-networkd with DHCPv4.SendOption and DHCPv6.SendOption for DHCPv4 and DHCPv6 servers respectively. For getting started into configuring your interfaces, I recommend to read the following resources:

I mostly used them for referential use in tandem with some other tools like systemd-networkd, not exactly informational cases. But still, I'm slowly grokking the documents if I have the time.

DHCP clients send some options that the DHCP server may accept or reject. These options include subnet mask, certain IP address, or the various URLs for the web server, SMTP server, etc. These may be used for more complex setups.

I've learned how to look at debug logs from the systemd-networkd.service especially that it can show the requests and responses of the DHCP client.

2023-01-23

I became frustrated as I clearly don't understanding about networking so I took a networking course from Google IT specialization on it for at least 2 hours. More specifically, on the Week 1 material. Because of this understanding, I also simplified the networking setup of my server which is relieving.

This misunderstanding basically leads to a distratrous chain of events such as possible misconfigurations with subnetting, Wireguard not working properly, and the private network not exactly configured as I would like to be in several interfaces. Ehhh, anyways, I've continued the course and it gave some more context but not enough. I'm continuing it all the way through from start to finish though despite looking for certain information to have a stronger foundational knowledge on networking. Really made me stumped for it.

2023-01-24

Here's the things I want to do today:

  • Continue my self-study courses
  • Properly configure my network interfaces with the services in mind in the private network
  • Administer my Keycloak instance
  • Install a self-managed instance of privacyIDEA

I continued the IT networking and the security course from Coursera today. The one I look forward the most is the networking course since it is the thing that gave me the most problems. I did study the concepts that I look forward to in the Week 2 material where it introduces the networking layer where IP started. The lesson is rapid in pace, introducing concepts and things related to networking per video. It does come with nice background reading in-between which sometimes introduces more things. Overall, I liked the introduction for what it is even if I knew some of them already from the networking wrangling done earlier. I look forward to the later modules of the course.

After that, I revised the networking setup which is the one that took the most time, unsurprisingly. Not only I discovered a network misconfiguration, I also discovered my own systemd-networkd is misconfigured.

The main culprit here is the ordering. systemd-networkd will load the units in alphanumeric order. Here's the setup for my server looked like before the fix:

/etc/systemd/network
├── 60-dhcpv6-pd-downstreams.network -> /etc/static/systemd/network/60-dhcpv6-pd-downstreams.network
├── 60-lan.network -> /etc/static/systemd/network/60-lan.network
├── 60-wan.network -> /etc/static/systemd/network/60-wan.network
├── 99-wireguard0.netdev -> /etc/static/systemd/network/99-wireguard0.netdev
└── 99-wireguard0.network -> /etc/static/systemd/network/99-wireguard0.network

This is supposed to configure my interfaces which is automatically configured for me. In this server, all hardware interfaces are Ethernet interfaces (e.g., ens3, ens10).

Since systemd-networkd applies them in alphabetical order, 60-dhcpv6-pd-downstreams.network is going to be applied first. Let's see what this network does.

[Match]
Name=en*

[Network]
DHCP=yes

[DHCPv6]
PrefixDelegationHint=fdee:b0de:5685:43ff::/64

Ohhh…

My intended configuration for systemd-networkd service is there are only two main nodes that are configured statically but all of them are applied with DHCP. This pretty much causes the supposedly statically configured networks to not be configured statically since they are already configured. The overall server is still working fine but not for internal servers.

Fortunately, the fix is simple: just reorder them and make sure that the statically configured IPs are applied first. In this case, just lower the number prefix (60) to something else (50).

As for considering having a self-managed instance of privacyIDEA, I'll do it after the tunneling service has been properly configured.

2023-02-01 - 2023-02-05

I just continued the self-studied courses from Coursera. That's pretty much it for the most part.

At 2023-02-04, I've solved most of the networking issues especially with DHCP. It is resolved by reducing the network interfaces connected to the server. The DHCP server seems to prioritize the IP addresses from the internal networks before the public-facing IPs which is why I mostly puzzled with this situation. On the other hand, networking setup in my server is also simplified which is always a plus to me.

I'll be considering using a DNS server (likely using dnsmasq) for my internal network. I don't know how to do this and I'll be continuing this for tomorrow session but I think I got the idea.

For future references, here's my tentative recipe:

  • Configure the DNS server to add names to a list of addresses.
  • Configure the reverse proxy to listen to those names.

As for considering the domain names, it will most likely go with .internal as the top-level domain. So, my Keycloak admin console web page should be accessible in auth.foodogsquared.internal, etc.

2023-02-06

Almost got tempted to look into another VPS provider since I got my first invoice from Hetzner today. I'm trying to look into cheaper solutions but it seems Hetzner quality services is worth it.

Anyways, I looked further into what services Hetzner provide and why I still chose it. For the cost of at least €9, I have: a decent server with sufficient cores (2 with at least 2.2 Ghz) and memory (4 GB which is definitely enough), a 1TB worth of storage, and other niceties. Those other niceties such as DDoS protection and free DNS servers bring it all together.

Speaking of DNS servers, I've decided to host a service but primarily for my own purposes. As I've seen from one of Julia Evan's posts about this exact question, apparently, you can host your own DNS server and pass the responsibility to secondary name servers of which Hetzner has… for FREE! Thus, making your primary authoritative server to be hidden. I can certainly take advantage of that but now I'm deciding what applications to use.

I've narrowed two answers so far: Bind and CoreDNS, both of which are available to be used as a NixOS module. Bind seems to be nice especially with its maturity and widespread usage. CoreDNS, on the other hand, also sees widespread usage especially with its integration with Kubernetes which is one of the reasons I'm considering it in the first place. Hard to choose but for now I'm scouting the tools. One of the main things I'm looking for is the ease of setting up the primary server to pass the records to "slave" servers which should be easy to find.

2023-02-07

Still cannot decide which application is which but being paralyzed to just two choices is one of the worst things you can do so I proceeded to test both. The first tool to be tested is CoreDNS simply because of its integration with Kubernetes.

It is very easy once I understood the syntax but its manual threw me off because of the how CoreDNS works with its plugins architecture which took me staring for minutes confused at where to look. Turns out all CoreDNS does is create a DNS zone which would then be configured by plugins which does the things that you may want to do.

With the plugins architecture, CoreDNS is very interesting to configure because in order to extend its functionalities, you need to include other plugins. Plugins, however, are embedded as in compiled within the executable and not in runtime which goes against my impression of a CoreDNS plugin (and one of the reason why I was off-guard with it). This unfortunately means you have to recompile CoreDNS just to include third-party plugins. I do wonder how this works with the NixOS which makes it (somewhat) easy with nixpkgs overrides but looking at the current Nix package definition, I think it will go smoothly for the most part. Not to mention, it is very easy to modify by changing the plugin.cfg. Thank you, nixpkgs overrides and CoreDNS for making it easy (at least)! Fortunately for me, there's no need to do some recompiling since all of the features that I want and I set out to do with a DNS server is within the built-in plugins anyways. I think I'll stay with CoreDNS for a little while. I'm convinced it will require less maintenance (configuration-wise) compared to Bind which already does a lot under-the-hood when looking at the service implementation at the respective NixOS module.

2023-02-08 - 2023-02-10

Two days of DNS server debugging. Pretty frustrating but I learnt a lot of things from that experience such as practically interacting with DNS queries from looking at the server logs and using tools such as dig and ss. This is where I my understanding with networking really ramped up built from the foundations I've learnt from the Coursera courses.

But experience mostly comes from debugging and solving problems. And there's a ton of them.

One of the problems is coming from misconfiguring your DNS servers which is frustrating to deal with the first time. This problem, however, is further split into tiny pieces. While I understood the basics of DNS (well, I think anyways), one of the pieces caused from misunderstanding how CoreDNS server blocks interacts with the system. More specifically, each server block attaches on every interface on your system when it listens which you can control them with the bind plugin. This way, you can control your internal network further with the appropriate server block.

I keep persisting that there should be separate server blocks for foodogsquared.one and plover.foodogsquared.one, not knowing that: 1) this will cause the server blocks to fight over to listen to the same interface, 2) this can be done on one server block. At the end, I opted for setting up the server with only one server block and assign it to listen it to all interfaces in the system explicitly with the bind plugin. It's much easier and sufficient to manage it in this way.

On the other hand, I found this DNS setup to be nice that I can easily manage both for the public-facing side of the server and the internal network. Though, I didn't set up the appropriate network configuration for the clients in the VPN tunnel: networking, routing, and all. At some point, I want to be able to get into the services with the domain names.

Another problem encountered is the firewall… Just had the wrong port for HTTPS makes my self-managed services to be inaccessible for a short while externally. I should have really seen this coming. I checked the logs from nginx and seeing no such access made it. There is also the fact that accessing the HTTP does work but redirecting it causes a timeout. It turns out it has been misconfigured for a long while. I just previously relied on the VPS firewall before since the resulting firewall configuration does not work for some reason.

There is also another thing special with this entry. It's the end of the journal. The original objective, after all, is to manage an external server for some time and learn a lot of sysadmin-related things along the way. I don't think I've managed to understand much of the required things in an average job listing. Instead, what I got is more foundational with the resources I eventually found. Not to mention, it is a plan after all. No plans are perfectly executed (at least in my experience).

I would summarize this portion to:

  • Managed a Linux server in the span of 3 months (practically 2.5 months considering I took a break in most of December).
  • Learned and dove into networking basics: the network layers and how they all work together. Some nice practical usage with the networking concepts with low-level tools such as setting up a DNS server, learning how networking is set in your system, and how applications are making use of these concepts.
  • Expanding more on the idea of deploying a system with Git and NixOS with infrastructure as code (IaC). It is very interesting in concept and worked well as a result. It is pretty fun especially with NixOS with its ability to easily interact with the deployments as generations. Practically, it means you can roll back the previous deployment if something goes wrong.

A lot of the concepts are foundational especially once I get into Kubernetes which I didn't touch on this span unfortunately. It seems like Kubernetes is pretty much an integration of all of the things I've used wrapped into a nice package: a toolset made up of smaller components that make up deployments similarly to systemd for being a toolset made up of smaller components that make up a system. Although I personally don't see much benefits especially that I use NixOS, it seems Kubernetes is not that bad. I'll have to try using it.

This will have to be done in part 2.

1

Where quality may vary but if you have no idea nor connections to start with, it's a good indicator as long as there's other more credible sources.

2

It's a basic fact, yes, but I haven't paid attention to these details yet.

3

Which I don't know if that's even safer but as long as the emergency is exclusively used for its purpose, I think it's safe.

4

They're also called dynamic and/or private ports.

5

Not supported for all providers, though.

6

An important detail is SSL is a predecessor of TLS and TLS is used as a replacement nowadays but most documents still refer to it. In other words, they are interchangeable.

7

Though, I'm not sure whether some services are appropriate for it to be hidden behind.

8

Just found it in the first page which is nice for me. Though, it is at the bottom on Google results while in the middle for Brave. I'm guessing because most Brave users are already tech-savvy thus tend to get higher quality when it comes to these types of topics.