foodogsquared/wiki
1
0
Fork 0
mirror of https://github.com/foo-dogsquared/wiki.git synced 2025-01-31 04:58:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
215ff303d9
wiki/notebook/journals.learning-how-to-sysadmin.org
Gabriel Arazas 215ff303d9 Update some journal entries for sysadmin journal 
Ehh... org-mode tables are neat.
2023-01-12 00:52:13 +08:00

736 lines
50 KiB
Org Mode
Raw Blame History

:PROPERTIES:
:ID: b0f0bb3f-9b8b-4035-b45f-021299918711
:END:
#+title: Journals: Learning how to sysadmin
#+date: 2022-11-10 14:14:04 +08:00
#+date_modified: 2023-01-07 01:16:29 +08:00
#+language: en
* Goals and expectations
While I've been managing a Linux desktop installation, it isn't the same as adminstering systems in a server setting.
So I want to learn by properly managing a server in some cloud platform (e.g., [[roam:Amazon Web Services]], [[roam:Google Cloud Platform]], [[roam:Microsoft Azure]]).
I would want to manage one (maybe with a [[roam:Debian Linux]] installation) in at least a quarter of a year and get to a level of getting a job out of it.
To be able to know the lay of the land, I'm able to find job listings from various sources [fn:: Where quality may vary but if you have no idea nor connections to start with, it's a good indicator as long as there's other more credible sources.].
The average list of skillset required seems to lean on the following:
- Knowledge in Linux components
- roam:nginx
- [[roam:Shell scripting]]
- Development deployment tools, mostly related to containers such as...
- [[id:9e4f04d4-00a3-4898-ac98-924957fa868b][Kubernetes]]
- roam:Docker (or roam:Podman)
- Knowledge in managing Windows servers
- Active Directory
- LDAP user and group administration
- roam:Powershell
This means I would have to start at 2022-11-10 and reach that level at least in 2023-02-10.
* Resources, preparations, and execution
The simplest way to get started is to maintain a server for a variety of purposes.
You could maintain a server for...
#+name: lst:example-services
- A password manager for your private accounts (e.g., Bitwarden or Vaultwarden).
- A game server for your friends (e.g., Minecraft).
- A code forge to store your own code (e.g., GitLab, Gitea).
- An audio server that can be used to play anywhere on your devices (e.g., Funkwhale, Navidrome).
- A content management system to serve your organization (e.g., Wordpress, Grav).
- A web server for your web applications.
- A chat server for your community (e.g., XMPP server, Matrix server).
- A social media instance for your community (e.g., Mastodon, Pleroma, Misskey, Lemmy).
In order to get started, you have to choose a virtual private server (VPS) host.
There are several of them to get started including the big 3 web services (roam:AWS, roam:GKE, and [[roam:Microsoft Azure]]), Digital Ocean, Hetzner Cloud, Linode, and vultr.
Each of them has a starting tier (with the big 3 has a free tier as long as you have a valid credit card information) that is under $5 dollars in average.
Alongside the [[lst:example-services][list of services]] that you might be interested in, there are a lot of applications and services to be interested.
- You could look into the services offered by [[https://yunohost.org/][Yunohost]], a Linux distribution that is primarily targeted to self-hosting services.
- There's a lot of services hosted at [[https://github.com/awesome-selfhosted/awesome-selfhosted][awesome-selfhosted/awesome-selfhosted]] from GitHub.
- Any services from [[roam:NixOS modules]].
Take note, each daily entry is made progress with 4 pomodoro sessions (2 hours) a day.
Then, the writing is done after, trying to recollect my experience as much detail as possible.
Anything more should explicitly be listed in the entry.
* Habits and todos
** DONE Learn kubernetes and Google Kubernetes Engine
SCHEDULED: <2022-11-10 Thu>
*** DONE Deploy a Vaultwarden instance
*** DONE Deploy an Archivebox service
*** DONE Deploy an RSS feed service
*** DONE Deploy a cluster
** TODO Manage a Linux server for at least 3 months
<2022-11-10 Thu>--<2023-02-10 Fri>
** TODO Manage a Windows server for at least 3 months
<2022-12-10 Sat>--<2023-03-10 Fri>
** TODO Deploy a NixOS image in virtual private server host
*** DONE Deploy with deploy-rs
*** DONE Deploy in Google Cloud Platform Compute Engine
*** HOLD Deploy in Microsoft Azure Linux VM
*** TODO Deploy in Hetzner Cloud
** TODO Self-hosted services
*** DONE Vaultwarden
*** DONE Gitea
*** HOLD Sourcehut
*** TODO OpenLDAP
*** TODO MinIO
*** TODO OpenVPN
* 2022-11-10
Started to journal my journey for system adminstering up to semi-professional standards.
For now, I'm scouting my options though I previously tried with Google Cloud Platform and deploying a [[id:9e4f04d4-00a3-4898-ac98-924957fa868b][Kubernetes]] cluster on it.
I might manage a Linux virtual machine right away using Compute Engine from Google Cloud Platform.
I'm very tempted to make it with a NixOS image as similarly laid out in [[https://ayats.org/blog/deploy-rs-example/][this blog post]] however I'm going with Debian as it is closer to traditional setup.
I may even consider something like Red Hat Linux or Rocky Linux.
* 2022-11-11
Retested the installation for cert-manager with [[https://cert-manager.io/docs/tutorials/getting-started-with-cert-manager-on-google-kubernetes-engine-using-lets-encrypt-for-ingress-ssl/][their page with Google Kubernetes Engine]] and I still didn't able to successfully complete using only a raw IP address.
Opportunity to buy a domain for myself and follow it the next time.
I'm very very tempted to make a NixOS deployment image as I've already seen what I can do with it but for now, I'll stick with the traditional tools.
However... managing both is an option. :)
* 2022-11-12
For now, I've been just managing a Debian virtual machine and successfully launched a publicly-accessible web server.
Mostly involves enabling the service for the web server and configuring the firewall.
It cannot be accessed easily since the instance's external IP is ephemeral.
As for letting HTTPS access, there is no such thing since signing certificates is only done with domains and I'm only accessing the server with a bare IP address.
This is the first time I have to worry about the lower-level things I haven't touched with the usual processes such as deploying websites and all.
Anyways in case you're curious, why Debian?
- It is stable.
Though, I have options such as deploying images with [[id:7e8e83d5-4b08-44f6-800d-a322f6960a62][NixOS]].
- Support and community is large.
It is a battle-tested distribution with a large package set and lots of resources have been written for this system.
- Consequently, it has [[https://www.debian.org/doc/][extensive documentation]] not only for beginners but also for various aspects like [[https://www.debian.org/releases/][its releases]].
While [[https://wiki.debian.org/][its community wiki]] is not as thoroughly documented as [[https://wiki.archlinux.org/][Arch Wiki]], it often contains enough information to get you by when managing a Debian server.
* 2022-11-13
I haven't done anything much in this day.
On the flip side, I'm overworrying about the price considering I'm in free trial and Google will only charge if I opt-in to activate the full account.
It turns out it isn't much of a worry if I leave it alone.
Having an ephemeral external IP address and being so low-value might have something to do with it.
For now, I'm going to plan to create a simple wiki server with the traditional LAMP stack.
What is it going to contain?
Simply my findings and mainly for configuring Mediawiki as I'm assuming the role of a sysadmin.
* 2022-11-16
Unfortunately, progress has stalled for now since I don't have a usable bank account for now.
Once it is available, progress should be quick with the availability of a domain name for me to mess around with.
A domain name is surprisingly affordable.
Just the services attached to it is where most of the expenses come from such as the domain email hosting and whatnot.
For the domain registrar, I picked Porkbun since it has a lot of sales and it is generally cheaper than something like Google Domains.
* 2022-11-22
Setup my own blog with the domain.
It was slightly confusing at this is my first time diving into the level of server and domain management.
First time encountered concepts like the DNS, CDNs, and managing DNS records, all of which I've learned from [[https://www.cloudflare.com/learning/][Cloudflare documentation]] of all things.
With the DNS management in place, I mostly learned how it interacts with the servers and makes it discoverable for other servers such as the effect of DNS caching which can take under a day to take effect.
The main problem I have encountered is redirecting my blog in https://foo-dogsquared.github.io into my custom domain.
However, I soon found out that all of the pages under my GitHub Pages domain is affected, making all of my project pages part of the domain.
The "easy" solution is just deploying my blog into a separate service and deploy the main GitHub Pages with a redirection page.
It is also the time I haven't used [[https://www.netlify.com/][Netlify]] in a long while.
The chain of problem never ends as now I would like to deploy my blog with Netlify easily.
Unfortunately, Netlify doesn't have an easy way to install and bootstrap an installation of [[id:3b3fdcbf-eb40-4c89-81f3-9d937a0be53c][Nix package manager]] unlike using with [[id:319b52f8-5e60-4bbf-b649-73d864ed186f][GitHub Actions]].
A solution for this would be using GitHub Actions to build and Netlify to deploy which fortunately [[https://github.com/nwtgck/actions-netlify][someone has already created a solution]].
Most of the problems I have are from misunderstanding and misconceptions of how DNS and server management works.
One of the prominent misconceptions I have is the DNS management is completely on the server, neglecting how clients can also affect the browsing experience.
This unfortunately took me two hours to figure out and completely missing the real problem.
Whoops...
There are some still misunderstanding with the DNS though.
I'll have to go with the basics.
I also thought GitHub Pages can be separated from domain per project page.
So that's another concept I didn't easily able to wrap around my head.
Despite the fumbling around, I would say not bad.
Now, I'm very very motivated to go self-hosting mode as I continue to host my [[https://github.com/foo-dogsquared/wiki][personal notes]] (that I continue to neglect updating).
I would like to self-host a Vaultwarden and Archivebox instance the next day.
# TODO: Illustration of Chain of encountered problems
* 2022-11-23
Another day, another time for DNS misadventures and misunderstanding.
This time, most of the problems come from the misunderstanding of how hosting works which is far off from my recent idea of a hosting provider where each part of different domains can be specified to make up the frontend of your website.
It turns out this is not the case.
I was able to deploy my blog with Netlify and set it to my domain.
Now, ~foodogsquared.one~ is open for the world!
I still haven't solved the issue of missing icons from the deployment but I'm very confident it is an DNS issue seeing as the "missing files" can be viewed, just viewed with the inappropriate headers that cause them to be blocked.
Not to mention, there is missing redirections for the old site which makes it inconvenient.
The only hope is nobody visits my site as I'm already dormant for the most services this year.
The thing only took about an hour where most of the time are spent in questioning and swatting the cache and tumbling over Porkbun's interface as I repeatedly reset and didn't realize my DNS records are kept being reset every time I want to point the domain to my GitHub Pages instance.
In any case, I'm just to going to delay fixing the issues from the blog site because I want to self-host some services. ;p
In this case, I want different services as part of one domain (e.g., my Vaultwarden and Archivebox instance under ~foodogsquared.one~).
It turns out that while Netlify allows some form of domain management, it simply isn't flexible enough especially since the services I put together are more likely to come from different sources.
I mean, simply deploying my blog already requires Netlify for it, what more for self-hosted services that Netlify cannot simply do?
To make it possible, I have to manage a proxy server that lets me sew in those services altogether under one domain.
That is, I want to access my Vaultwarden instance in ~vault.foodogsquared.one~, I want my feeds to be accessible in ~feeds.foodogsquared.one~, I want to self-host my code in ~forge.foodogsquared.one~ among other examples.
Luckily for me, several of them exist such as [[https://www.nginx.com/][Nginx]], [[https://caddyserver.com/][Caddy]], and the good 'ol [[https://httpd.apache.org/][Apache HTTP server]] all of which listed software have capabilities beyond a simple server.
However, I chose Nginx seeing as it is the popular tool in hand and also because a lot of job listings that list knowledge for nginx as part of their wishlists.
Currently, I gave in to the temptation to configure all of the servers with [[id:7e8e83d5-4b08-44f6-800d-a322f6960a62][NixOS]].
Alongside the fact that I already have enough for imperatively managing them servers, there are additional three main factors to this decision:
- The declarative configuration.
- A framework for generating custom images ([[https://github.com/nix-community/nixos-generators][nixos-generators]]) that is built on top of [[id:f884a71c-0a0f-4fd7-82ff-00674ed4bd66][nixpkgs]].
- The fact I already have [[https://github.com/foo-dogsquared/nixos-config/][an existing configuration]] that can serve as a framework to easily instantiate individual nodes.
The hardest part is creating my first image which is going to be deployed in Google Cloud Platform.
The second hardest part is managing my Google Cloud Platform account as the mountainous amount of things I have to keep in mind whenever I'm staring at the dashboard of several cloud providers.
The third hardest part is the amount of prerequisites before I even start doing one thing which is already a thing that the second hardest part is giving me.
Unfortunate...
On the other hand, my NixOS configuration is slowly turning to be a nice monorepo for deploying everything I want.
It is surprisingly easier to manage them but the part that's giving me the hardest part is the deployment.
As for private files and deployments, this is easy to manage with [[https://git-scm.com/docs/git-worktree][Git worktrees]] which is somewhat tedious to make sure my public and private branches to sync.
* 2022-11-24
The configuration for Vaultwarden are in place in my first NixOS-powered deployment but most of the problems are from the lack of understanding the networking infrastructure.
Fortunately for me, there is the [[https://www.debian.org/doc/manuals/debian-handbook/][Debian Handbook]] with details on each facets on the infrastructure.
It is specifically aimed for Debian systems but it is good enough if you're familiar with the interface (which is just a command-line shell such as Bash).
Before that, the trouble comes from setting up a mailer which is troublesome if you only have a GMail account.
However, I'm also considering to move my email provider from GMail to something else.
Candidates include Fastmail, Zoho Mail, and mailbox.org, all of which has a paid plan (and also a long trial period of at least 2 weeks).
In the end, I decided to not use mailing services altogether for my self-hosted services for the time being.
As for self-hosting my code, I did initially consider Sourcehut since I'm largely interested in how much resources it needs to host it.
However, that didn't work out as there seems to be a lot of maintenance required for my current needs which is simple right now.
I still heavily consider it for future endeavors though especially with its comprehensive documentation and integration of services is just nice to have.
Not to mention, Sourcehut is still in alpha which indicates the maintainers still have plans for it.
Its primary maintainer especially [[https://news.ycombinator.com/item?id=31964064][considers Sourcehut to be easier to self-host]] so the plan of self-hosting Sourcehut is not entirely thrown away.
In the end, I decided to use [[https://gitea.io/][Gitea]] considering there is already a NixOS module for it (at least in version 22.11) and implementing a new way to communicate between forges with [[https://forgefed.org/][ForgeFed]].
This means collaboration between different instances is very much possible and I'm in support for them.
Compared to Sourcehut, Gitea is simpler to initialize which I was able to quickly start an instance.
Most of the time came from viewing the configuration options and testing the instance.
* 2022-11-26
The deployment failed because I forgot secrets management is a thing.
Each infrastructure-as-a-service apparently have their own thing such as [[https://cloud.google.com/kms][Google Cloud Platform KMS]], [[https://azure.microsoft.com/en-us/products/key-vault/][Microsoft Azure Key Vault]], [[https://aws.amazon.com/kms/][Amazon Web Services KMS]], and [[https://www.hashicorp.com/products/vault][Hashicorp Vault]].
It's a good thing I'm using [[https://github.com/mozilla/sops][sops]] for this.
It's a short time from the previous days but it should go back to normal with the time to journal this abomination.
Around this time I also signed up for Microsoft Azure free tier subscription for trying to manage a Windows server this time around.
It should be simpler to start since I have absolutely no idea how to provision a thing but compared to my knowledge for Linux-based systems, it is non-existent.
So most of the time spent will be learning all of the concepts from absolute zero experience.
Should be fun...
I've also decided to full gung-ho on deploying Linux-based systems with NixOS.
I've deleted all of the non-NixOS Linux-based systems in my fleet and started generating a bunch of NixOS GCE images.
Should be doubly fun...
* 2022-11-27
For now, I've gone back to managing a deployment with Linux-based systems and try to *properly* manage this time around.
For a start, I decided to manage the static websites separately from the server since [[https://answers.netlify.com/t/support-guide-why-not-proxy-to-netlify/8869][Netlify apparently does not bode well to proxies]].
At the very least, those websites can now go at their own pace instead of deploying them altogether.
Second, most of the services are misconfigured.
Classic...
Most of the domains and settings are not properly configured which means I have to review the documentation for the nth time.
It's not exactly a chore especially that this is my first time managing all these things.
Not to mention, NixOS surely does some things differently sometimes which does not go well with me especially that I rely on resources that are mostly written with the mainstream distributions in mind (i.e., Debian, Ubuntu).
Finally, I'm now going to add one more component into my server which is [[https://www.postgresql.org/][PostgreSQL]].
All of the services I've used so far can be configured to use SQLite which makes things easier but SQLite is mainly made with the filesystem in mind unlike PostgreSQL which is primarily made with network services.
Fortunately for me, [[https://www.postgresql.org/docs/][its documentation]] is easy to follow.
At least for tomorrow, I plan to manage one more component into the mix with an [[https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol][LDAP]] server for user and group management which has a lot of presence in job listings that I've seen.
Fortunately for me, there is an [[https://www.openldap.org/][OpenLDAP]] service module already available in NixOS.
I just have to be careful in chewing in managing this seemingly simple server.
* 2022-11-28
Welp, most of the configurations of the services should be fixed but the last thing that remains is proper deployment with the secrets.
While I could do that by simply transferring the private key into the virtual machine, it just misses the point of using a key management system which GCP already has.
Pretty much, I'm missing out on it if I didn't use it so I have to use it. :)
From what I can understand, with sops, you have to set the [[https://cloud.google.com/docs/authentication/application-default-credentials][proper credentials]] to be able to decrypt it.
That's fine for local development environment but it isn't nice for deployed systems.
One of the ways to properly set it is by using [[https://cloud.google.com/docs/authentication/provide-credentials-adc#attached-sa][a service account]] with the proper permissions which in this case for encrypting and decrypting GCP KMS keys.
So I created a user-managed service account to be used for the server, set the proper permissions, and [[https://cloud.google.com/iam/docs/impersonating-service-accounts][make the user-managed account impersonate as the default service account]] because I don't want to manually manage that.
Be sure to read up more on [[https://cloud.google.com/iam/docs/best-practices-service-accounts][how to properly manage service accounts]].
The reason why I laid it all down in this writing is because the documentation of Google Cloud Platform is surprisingly nice to use... sometimes.
The way they show different ways to accomplish a task with different tools (e.g., Console, ~gcloud~) is a nice touch.
However, the amount of looping links makes it easy to get overwhelmed.
Am I the only who just repeatedly visit between different pages just to get the idea from a single page?
I understand the reasoning as a knowledge base that caters both to new and experienced users but it is something that I experienced.
I feel like the process of simply doing those steps previously mentioned should take way shorter time than it should be.
Most of the time is spent in staring at those pages, trying to see if I'm following it right.
This is where I feel like I should've really started with [[https://go.qwiklabs.com/][Qwiklabs]] which I didn't is a thing when I started.
Welp...
* 2022-12-01
Here we go, start of December.
Only two months to go before the deadline to become halfway to professional-level (or at least getting paid).
I haven't done anything from the last two days so there's no entry for it.
Still having some problems, mainly from PostgreSQL service this time.
I'll use this opportunity to experiment debugging and maintaining services with PostgreSQL.
Thankfully, its [[https://www.postgresql.org/docs/][documentation]] is very comprehensive especially that it has a dedicated chapter for server management.
I'm only starting to wrap around my head with the concepts of a database and its management.
The errors from the database service are more likely from the lack of proper privileges.
From the Vaultwarden service, the new error this time looks like this.
#+begin_src log
Dec 01 01:41:03 localhost vaultwarden[762]: [2022-12-01 01:41:03.533][vaultwarden::util][WARN] Can't connect to database, retrying: DieselMig.
Dec 01 01:41:03 localhost vaultwarden[762]: [CAUSE] QueryError(
Dec 01 01:41:03 localhost vaultwarden[762]: DatabaseError(
Dec 01 01:41:03 localhost vaultwarden[762]: __Unknown,
Dec 01 01:41:03 localhost vaultwarden[762]: "permission denied for schema public",
Dec 01 01:41:03 localhost vaultwarden[762]: ),
Dec 01 01:41:03 localhost vaultwarden[762]: )
#+end_src
The error is a bit intuitive with the intuition being a permission error with the 'public' schema.
I've simply resolved the error by adding the permissions from the NixOS config like the following snippet.
#+begin_src nix
{
services.postgresql = {
enable = true;
ensureDatabases = [ vaultwardenDbName ];
ensureUsers = [
{
name = vaultwardenUserName;
ensurePermissions = {
"DATABASE ${vaultwardenUserName}" = "ALL PRIVILEGES";
"ALL TABLES IN SCHEMA public" = "ALL PRIVILEGES";
};
}
];
};
}
#+end_src
As an additional fact, I've quickly come across from the documentation that [[https://www.postgresql.org/docs/15/ddl-schemas.html#DDL-SCHEMAS-PUBLIC]['public' schema is the fallback schema for databases without names]].
That's something to keep in mind in the future.
But anyways, here's a light writing on the summarized version of my understanding of the database starting with its authorization process, the part where I'm spending the most time on understanding it.
#+begin_quote
Being accessible through different ways, widely available to other users, and globally contains various application data, the PostgreSQL service has ways to make sure access to the database is only done by trusted users.
This is where authorization comes in similar to POSIX-based systems when authorizing access to various services.
Inside of the database, various services (which serves as clients) want to access their data which the database contains a variety of them.
In order to safely access them without much problems, PostgreSQL plants some ways to verify its client.
There are different ways PostgreSQL can authorize access to different users.
- Tried-and-true password authentication for the user it tries to access as.
- LDAP authentication.
- Another way is simply leaving a map of connections and their trusted users in a file (e.g., =pg_hba.conf=).
- Accessing the database as one of the user of the system with the same name as the user of the database.
This makes sense: it is more likely a dedicated user specifically created for a certain service alongside a database for that service.
This authorization is referred to as an *ident authorization*.
Several examples include hosted services with dedicated setup (e.g., user and user group, database) as they're logically mapped from the operating system and its different components.
#+end_quote
As for the plan to maintain an LDAP server and user management with it, I'll start around this week hopefully.
For now, the focus is debugging and maintaining a server.
Mainly, by SSHing into the server and getting used to the maintenance tools with [[id:20830b22-9e55-42a6-9cef-62a1697ea63d][systemd]].
There are also some things I've learned such as:
- Creating a new unit file easily with ~systemctl edit --full --force $UNIT~ and it will just place it in one of the unit paths.
- Easily viewing how much journal files took space with ~journalctl --disk-usage~ which also supports it at [[id:c7edff80-6dea-47fc-8ecd-e43b5ab8fb1e][systemd at user-level]] with =--user= flag.
- Flushing all ephemeral journal files from =/run= to a persistent storage with ~journalctl --flush~.
- Log rotation with ~journalctl --rotate~.
- Ports lower than 1024 is a privileged port and normal users cannot use it. [fn:: It's a basic fact, yes, but I haven't paid attention to these details yet.]
I'll get around to using a traditional Linux distro (Debian, again).
While server management with NixOS is nice, I think getting used to the traditional environment nets more credibility.
Though, it is getting easier to map concepts I'm getting from NixOS to the traditional setup with time.
Especially that most of the things from NixOS are setting up services which could be done in any Linux environment anyways.
While I'm at it, I'm starting to look into backup services.
I'm already using [[https://borgbase.com][Borgbase]] with only the free 10 GB but it quickly ran out.
For now, I'm looking for a cheaper option if there's any.
* 2022-12-02
Hey, the Sendgrid account application has been approved.
Well, that's one more service component to enable.
There's also one very stupid mistake I didn't realize I'm making at this point from the very beginning: I keep hitting the [[https://letsencrypt.org/docs/duplicate-certificate-limit/][duplicate certification limit]] and I keep forgetting to backup the certificates.
I KEEP DOING THIS FOR THE PRODUCTION DEPLOYMENT, WHAT THE HELL!
I always thought the production only makes the same error as the development deployment.
Welp, I'll just wait for a week then everything should be fine and dandy for the production.
I'll just use test environments with domains with test subdomains on them while I'm at it.
Or I could just generate a new request with a different "exact set" of domains as already mentioned from the Workarounds section from the aforementioned page.
* 2022-12-03
Today's theme for misconfigurations are permissions.
Permission error for inaccessibility for PostgreSQL schemas for a certain user, permission error because the users does not have the permission to access the files, et cetera, et cetera.
Though, we're now in the home stretch without much shenanigans.
I'm now slowly chewing all of the things I bit off.
On the other hand, I found out about how certificates are generated in NixOS with the default workflow.
It is using [[https://go-acme.github.io/lego/][lego]] which has support for [[https://go-acme.github.io/lego/dns/][tons of DNS providers]] including Porkbun which is the domain registrar I used to buy my domain name.
Nice!
Now, there is more automation including taking care of setting the appropriate DNS records and now the shenanigans with certificates is on the past (or at least significantly reduced).
All I have to take care of is my secrets file and managing my servers with some security which I created the following initial guidelines as a start.
- Starting with removing the keys from various KMS such as from Google Cloud Platform.
This makes it easier to decrypt keys if someone has access to one of the users from its respective virtual machine instances since all of them are configured with their respective KMS enabled.
So they have to go.
- Giving some more thought about managing secrets with their respective keys.
Some examples include giving the least privileges with the minimal number of keys plus a fallback key for emergency. [fn:: Which I don't know if that's even safer but as long as the emergency is exclusively used for its purpose, I think it's safe.]
Also, while taking care of that, I also found out about [[https://kb.porkbun.com/article/190-getting-started-with-the-porkbun-api][Porkbun API]] which means I could create a simple program to interact with my DNS records instead of going to the website.
There are still some things to set up in the website but that's not much of a problem.
* 2022-12-04
Took a more serious approach to learning PostgreSQL this time because of primarily two reasons:
- Apparently, there are [[https://www.postgresql.org/docs/15/release-15.html#id-1.11.6.6.4][changes related to the =public= schema]] which is where most of the problems are from.
That is indeed something to keep in mind as [[2022-12-01]] has already mentioned.
- I only haphazardly glanced of the documentation before.
Really, the approach is more scattered: just debug the errors from the services and go to the appropriate chapters.
Which is fine for the most part but it can completely screw me over if I'm not mindful of the concepts.
With the listed information in mind, I deduced the problems are really coming from the applications to use the default schema which is in =public= (which is problematic for my configuration).
But I missed a very *very* crucial information that made me bash my head to this simple problem for hours over the previous days.
Apparently, most of them services can be configured to use another schema.
Yeah.
That's it.
What the hail...
However, there are consequences that made me understanding a bit how to properly manage a PostgreSQL database.
Specifically, what I've done after is trying to understand is how permissions and other objects are handled through the entire clusters.
More specifically: schemas and roles which my understanding is proven to be lackluster.
With the new understanding, I came across a new way to manage them schemas.
The maintainers of PostgreSQL encourages a more secure usage pattern for managing schema referring to as *secure schema usage pattern*.
In the encouraged practice, database users have their own schema (with the similar name for their schema, of course) and encouraged to set the schema search path to be set only for the user themselves (e.g., =search_path = "$user"=).
Then, with the applications lending a setting to configure the schema, you could make use of the appropriate schema for the appropriate user for that certain service.
I can see what the maintainers are encouraging here.
With the different ways to authenticate (especially with the ident authentication and the like), each service which may be running with their corresponding operating system user has a corresponding database user that owns a corresponding schema for the service data and whatever objects they want to access.
It's a nice, easy-to-understand practice that easily maps between the operating system and the database objects.
With the secure schema usage pattern, you're essentially playing Match-3 for the sake of simplifying the system.
Anything that reduces cognitive load for that is good enough for me.
On the other hand, the workflow for deployment NixOS-based systems is great with [[https://github.com/serokell/deploy-rs][deploy-rs]].
Though, the only issue is [[https://github.com/serokell/deploy-rs/issues/78][I cannot deploy the config with user passwords]].
The terminal is usually mangled up while using it with the SSH'd process.
Thankfully, [[https://github.com/serokell/deploy-rs/issues/78#issuecomment-989069609][one of the comment has shown a solution by disabling magic rollbacks]] which is unfortunate as I really would like to have those but it works.
There is [[https://github.com/serokell/deploy-rs/issues/78#issuecomment-1133054583][another comment that shows what's really happening under the hood]] which I can confirm with the part about how the input is echoed and fits with my experience of seemingly janky input with the mangled output.
Another quick solution is configuring the SSH user to be passwordless with sudo which is not great to me.
Don't like passwordless sudo especially that I've set a password for the user, making it essentially useless.
* 2022-12-05 - 2022-12-07
Nothing much happened here aside from sporadic debugging sessions.
I took this opportunity to learn how to debug the system mainly with [[id:20830b22-9e55-42a6-9cef-62a1697ea63d][systemd]] and PostgreSQL components.
More specifically, some new things found:
- I found how great =systemctl show= subcommand is.
You can view the properties of a unit which is nicer for debugging [[id:cd5f0d04-d9bb-44e8-a0f2-630ea58c1e94][systemd services]].
Not to mention, you can specify the properties with the =-P= flag (e.g., =-P User=, =-P Group=).
- Some facts with networking ports.
This is not something I managed over the days since Google Compute Engine firewall has an easy setup.
- Ports under 1024 are considered privileged (or system) ports and normal users cannot use them.
Ports on 1024-49151 are registered (or user) ports and are maintained by IANA to assign and map ports to services.
Ports on 49152-65535 are private ports and often ephemeral. [fn:: They're also called dynamic and/or private ports.]
- There are [[https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers][conventional port numbers used for certain services]] especially in privileged ports.
- [[https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml][IANA does have a registry of port numbers]] for services with databases in different formats available for download.
* 2022-12-08
TL;DR - I've looked into managing emails with my system which is a responsibility for administering systems, yeah?
Anyways, I've been looking for an alternative email hosting provider that is not Google and found out about mailbox.org from [[https://www.privacyguides.org][Privacy Guides]].
[[https://mailbox.org/en/post/christmas-voucher-2022][They happened to have an annual Christmas campaign]] which nets me free usage of their standard plan for six months.
Talk about timing.
What makes mailbox.org odd to me is the amount of services they provide and its pricing.
For €3 (about ₱180), you have basically the equivalent of Google Workspace suite including 10 GB of space for email (which is huge for me excluding mailing list discussions) and 5 GB of cloud storage which is not much but it is nice.
I would overlook mailbox.org's other services and only in it for the email provider buuuut I have found some neat things about their services.
I first misjudged that it would be just some low-quality office suite services with the email service as the forerunner but it apparently uses [[https://www.open-xchange.com/][Open-Xchange]] (OX) which means it can work with OX-compatible apps.
Not to mention, their interface and infrastructure are built on it which [[https://mailbox.org/en/post/ox-summit-2019-in-madrid-mailbox-org-and-enterprise-partners-meet][the maintainers of the service are also involved with the development of the OX app suite]].
Aaaaand that opened up the world (or rabbithole) of office suites to me, out of all things.
Aside from the services, there are some neat things with the email service itself.
One such example is [[https://kb.mailbox.org/en/private/e-mail-article/ensuring-e-mails-are-sent-securely][secure emails]] which is sent through an email address through =secure.mailbox.org=. [fn:: Not supported for all providers, though.]
Sorry for misjudging mailbox.org as it turning out to be a solid provider the more I look into it.
It turns out what mailbox.org provides is an office suite, basically an all-around alternative to Google Workspace.
I can see myself being their customer for years though I'm mostly feel like it's a waste especially I'm looking only into email hosting.
However, it is nice for a non-Google Workspace/Microsoft 365 setup if you want a more privacy-oriented workspace suite for your family or group.
At any rate, there are some services that are exclusively focused on email hosting such as...
- [[https://mxroute.com/][MXroute]] which has a nice option considering you have unlimited domains with unlimited email addresses.
It costs $45 for a year which is cheaper compared to other services but it isn't for me (and I just missed the Black Friday sale, CRAP!).
Comparing this option to my current needs and situation (only using one domain and using mailbox.org), it isn't the suitable choice considering the price is slightly close (without sales discount or anything).
- [[https://purelymail.com/][Purelymail]] is another service that is purely focused on email.
As it motto says, "Cheap, no-nonsense email".
It only costs $10 a year and already allows custom domain
One criteria I left out in the list is the spam filtering.
Ideally, it should be done on the service provider side which is present on all listed alternatives so far.
The question is how active and the quality of the spam filtering service.
MXroute seems to be more on the active side and reportedly has more positive review especially with the way how emails are sent through their service.
As for Purelymail, I don't see much things in that side.
At this point, MXroute is going to be my secondary choice for email hosting.
For now, mailbox.org is enough for my needs.
Both services required to be configured for sensible spam filtering anyways and I am currently under free credit anyways so I should have time to know about some details about email service in general.
There is also little trivia I've learned specifically with email aliases.
In mailbox.org, there are multiple email aliases that can be acquired for free (e.g., =postmaster@ADDRESS=, =hostmaster@ADDRESS=, =webmaster@ADDRESS=, =abuse@ADDRESS=).
This is apparently a convention for it, specifically [[https://www.ietf.org/rfc/rfc2142.txt][RFC2142]] that highly encourages certain aliases for different parts of the services provider.
Here's some examples from the document:
- =abuse@= is for public inappropriate behavior.
- =webmaster@= is for handling HTTP but practically it's the account for handling issues in the website.
- =postmaster@= is for SMTP, typically used for mail services.
* 2022-12-10
The server is mostly operational but it required some non-declarative setup beforehand.
Not exactly close to what I'm aiming for but it is close enough.
Really, the problems mostly comes from the way PostgreSQL 15 handles the schema which I would like to take advantage not only with the latest improvements on the package but also its recommended practices.
It seems like the current service NixOS module is not modelled after those practices.
The practices I tried to apply are highly encouraged in version 15 especially with its changes.
The NixOS module tries to cater for the majority of available versions.
On the other hand, I've configured my server to have backups for anything appropriate though most of the focus is on application data.
Most of the services have a documentation for dumping data like in [[https://github.com/dani-garcia/vaultwarden/wiki/Backing-up-your-vault][Vaultwarden]], [[https://docs.gitea.io/en-us/backup-and-restore/][Gitea]], and [[https://www.postgresql.org/docs/15/backup.html][PostgreSQL]].
All I have to do at this point is to back them all up with [[https://www.borgbackup.org/][Borg]] which I have a remote backup hosting at [[https://www.borgbase.com/][Borgbase]].
It's seriously NICE to have the configuration coming together in a neat little package.
Aaaaaaand the LDAP server is becoming more of an afterthought.
Aaaaaaand speaking of afterthought, I'm considering to host a VPN service for my own connections.
On yet another hand, I've been considering about the VPS hosting provider, preparing to move away from Google Cloud Platform.
It is a shame as the platform is a nice tool.
I easily created an automated workflow that'll deploy my existing configurations in the cloud somewhere especially with [[https://cloud.google.com/sdk/gcloud][gcloud]] tool.
I've considered [[https://hetzner.com/cloud][Hetzner Cloud]] as the VPS host lately as the server options is ridiculously cheap.
About €5 for a complete server with a public external IP compared to about [[https://cloud.google.com/products/calculator#id=227e202d-86cb-478b-94d4-296ccc72f805][$30 in Google Cloud]].
It also has a [[https://github.com/hetznercloud/cli][command-line utility humorously named =hcloud=]] which I should be able to easily automate it.
* 2022-12-12
The test Linux server is now mostly operational.
It is nigh time for managing a Windows server and make them communicate with each other, probably with services with Active Directory which is apparently an LDAP server with Windows-specific bells and whistles.
Today, I've also learned about the existence of =systemd-tmpfiles= for customizing Gitea from my NixOS configuration.
However, there are some things left to do in this production server in development.
Most of them are involved with authentication services.
- For one, I would surely create an LDAP server just for the kicks, specficially kick up an [[https://www.openldap.org/][OpenLDAP]] server.
- I've decided to add one more authentication service, mainly for the web with single sign-on and social logins.
This is apparently a separate service to an LDAP directory service.
For this, I decided to host with [[https://www.keycloak.org][Keycloak]].
- Add the appropriate settings for the already existing services.
Since the additional services listed here require more caution for using it, we may as well make it for the rest of the system.
This includes adding [[https://www.postgresql.org/docs/15/ssl-tcp.html#SSL-CERTIFICATE-CREATION][secure TCP/IP connections with SSL]] for my PostreSQL service.
Aside from these authentication services, I reviewed my understanding of CA certificates in relation to HTTP(S).
As HTTP is stateless, it isn't concerned who is communicating between each endpoint.
However with HTTPS, it is a different story.
HTTPS is essentially HTTP + SSL/TLS. [fn:: An important detail is SSL is a predecessor of TLS and TLS is used as a replacement nowadays but most documents still refer to it. In other words, they are interchangeable.]
From what I can understand in TLS, the certificates are composed of a keypair: one issued as a public key for clients to communicate with the server and the private key to verify them so the server can communicate with the client securely.
# TODO: Workflow illustration of the SSL for applications
In order for applications to make use of this, you can either configure them to point the certificate files.
This is commonly used for web servers (e.g., Nginx, Apache, Caddy).
There are also other applications that make use of this such as databases (e.g., PostgreSQL, MySQL), [[id:9e4f04d4-00a3-4898-ac98-924957fa868b][Kubernetes]], and authorization services (e.g., Keycloak).
* 2022-12-15
Today's theme for management is: secrets management.
While this is already done for my NixOS setup which is done with sops, keeping those secret keys is now a matter of securely keeping it.
To solve this problem, we have to lay out all of the information of our current situation:
- There are private keys for different formats: GPG, SSH, and age.
Not to mention, remote secrets such as from GCP KMS, Azure Vault, Hashicorp Vault, and AWS KMS.
- Proper storage for these keys.
This is especially important for GPG where it revolves around your identity.
As I don't have an iota how to do it *right*, I followed [[https://alexcabal.com/creating-the-perfect-gpg-keypair][someone's guide for this]] instead.
More specifically, I followed the recommended resource from that post which is from [[http://wiki.debian.org/subkeys][the subkeys management page from the Debian Wiki]].
- Multiple keys management.
I want to properly learn how to manage them keys for different purposes.
- Backing up properly which is already done with borg.
Hoorah for me...
* 2022-12-18
Now, the start of properly setting up them authorization services.
More specifically: Keycloak and OpenLDAP.
- For OpenLDAP, the project has a nice [[https://www.openldap.org/doc/][documentation]] for configuring and administrating an LDAP server.
I practically have no choice since other alternatives such as FreeIPA and Authelia is not exactly present as part of the available NixOS modules.
- For Keycloak, the [[https://www.keycloak.org/documentation][documentation]] has a nice structure to it including references and "Getting started" guides.
Though, all of the time are spent on learning OpenLDAP instead.
Looking at the examples, it is quite verbose.
It is becoming similar to the Google Cloud Platform documentation problem where it is becoming overwhelming and requires through pages back and forth.
Don't get me wrong, it is nicely structured but it is verbose.
I think I need some more time to absorb this.
Even if I skimmed it, I cannot get a bigger picture clearly.
Looking at the whole picture, it is pretty simple as to what it is.
Most of the details are hidden behind conventions which is where my problems lie.
It really requires familiarity which is acquired with time.
Not to mention there's not much examples I can see in the whole wide web so I cannot freely experiment some things.
(Thank good God for NixOS enabling easy experiment with building VMs easily though.)
I'll put up with studying about Keycloak later.
While I can see myself using Keycloak more, LDAP seems to be more common judging from the job listings which is why I'm studying this in the first place.
Active Directory being one of the most prominent example for this.
Speaking of which, I should really get with managing a Windows server at some point.
Preferably, they would have to be done in early January as I'm not done configuring the (NixOS) Linux server yet.
* 2023-01-01
Took some holiday vacation.
Back to normal scheduling.
I've been deciding on migrating host to Hetzner Cloud since it is way cheaper compared to Google Cloud Platform.
The disadvantage is there's no way to make into provisioning with a custom ISO except with a kexec-based image.
However, you can easily initialize a NixOS system with [[https://github.com/elitak/nixos-infect/][nixos-infect]], a script that converts a traditional Linux system into NixOS.
That's pretty much it for the most part.
Most of the things done is only for migrating and getting familiar with Hetzner Cloud.
It is pretty light on the thing but most of the system are not working due to ACME.
I'll just wait for some time and tinker with it some more.
* 2023-01-04
I'm now trying to get a concrete step into showing some credentials by completing certain courses in Coursera.
It is mostly for certificates just to show that I have self-drive.
Fortunately, Coursera has a financial aid initiative which is quite easy to get approved of but it does take some time.
For now, I've been completing a part of the [[https://www.coursera.org/professional-certificates/google-it-support][Google IT Support Professional Certificate]], mostly in the security side as it is the weakest side that I know.
I'm also taking the [[https://www.coursera.org/specializations/cloud-native-development-openshift-kubernetes][Cloud-Native Development with OpenShift and Kubernetes specialization]] also for the certificate but learning more about Red Hat systems seems to be interesting.
Fun and beneficial: double the benefits!
For now, the self-managed server has been stopped since I'm just waiting for the certificate cooldown.
The server should be backed up a couple days from now.
* 2023-01-06
The server is back up and running!
I also added an Atuin sync server especially that I use more often now and slowly needing to sync the history.
While the sync server from the developer is nice, I decided at the end that I would like to manage it for a try.
Fortunately, the service is already available as a NixOS module so there's not much problem there.
Next up, I also replaced the OpenLDAP server to be managed with [[https://github.com/majewsky/portunus][Portunus]] which is already available as a NixOS module.
I haven't tried completely managing it yet, though.
I only scratched the very surface of managing and using LDAP.
I should find out more about it once I continued with my courses I previously mentioned in this journal.
There is another thing which is adding a new component in my server: a nice storage box.
Specifically, I added a storage box from Hetzner which is surprisingly cheap with €4 for a terabyte.
What's more, this storage box has some neat features including an SSH and Borg server.
This instantly replaces my decision to go with a Hetzner storage box instead of Borgbase which is nice but with the budget in mind, I get more mileage from Hetzner's offer.
A complete self-managed setup for €8 a month?
That is very nice especially with a budget.
I'm very satisfied with Hetzner so far.
Even with a meager job pay, my whole setup is quite cheap.
Here's a table of the expenses.
| Thing | Expenses (in EUR per month) |
|--------------------------+-----------------------------|
| Hetzner VPS | 5.35 |
| Hetzner storage box | 3.5 |
| mailbox.org account | 3 |
| foodogsquared.one domain | 2 |
|--------------------------+-----------------------------|
| Total | 13.85 |
#+TBLFM: @>$2=vsum(@I..@-I)
Reference in New Issue View Git Blame Copy Permalink