Compiling MentOS: Mentoring Operating System

For the Operating Systems lab, it has been requested to start to autonomously compile MentOS1, Mentoring Operating System, a “hobbyist” operating system developed by some students of the University of Verona. Unlike many other operating systems used in other courses, MentOS is among the few to dogmatically follow the guidelines made available by Linux, implementing the same data structures and algorithms.

This article is intended to be a sort of step-by-step guide to compiling MentOS. The sections to follow all depend on the operating system host where you are going to compile MentOS.

For Ubuntu/Debian

Before proceeding to install the packages for MentOS, let’s go ahead and update any packages already installed:

sudo apt update && sudo apt upgrade -y

To compile MentOS we will need some packages like build-essential (package that includes GNU compilers for C and C++), git (for the versioning system), cmake (build system), qemu-system-i386 (to create a virtualized machine) and doxygen (for documentation generation). The -y flag already assumes the YES choice during installation.

We install the packages through apt:

sudo apt-get install -y build-essential git cmake qemu-system-i386 doxygen
In the case of operating systems other than Ubuntu/Debian, but still Unix-based, you can replace apt with your preferred package manager or one already installed on the host machine. For example on Arch, the base-devel package should suffice.

For those who are also interested in debugging MentOS, the developers have implemented a very fast and simple system that relies on the use of cgdb and xterm.

sudo apt-get install -y cgdb xterm


For MacOS, package installation is a bit more difficult than Unix-based installation: MacOS does not have a package management system. To make up for this lack, some developers have created Brew, an unofficial package manager.

As for the Unix-based section, we proceed to update the package list with the command:

sudo brew update && brew upgrade

We install the mandatory packages for the compilation of MentOS:

sudo brew install i386-elf-binutils i386-elf-gcc git cmake qemu nasm

If we want to debug, we also install cgdb and xterm:

sudo brew install cgdb xterm

“I don’t like Brew/I want to try the thrill of compiling gcc myself”

If you don’t like Brew and want to try the thrill of compiling gcc yourself on a macOS platform, congratulations on being brave. Follow the instructions in the article compiling the toolchain on macOS.


For all operating systems (except Windows, not considered in this guide), the steps to compile MentOS are the same.

We run git clone to get the MentOS source code (<clone_directory> is the folder where we want to get the sources):

git clone <clone_directory>
cd <clone_directory>

We create a new folder called build that will be used by CMake to write configuration files for building and compiling.

mkdir build && cd build
cmake ..
If we encounter errors during the process, don’t panic. Most of the time it’s a problem with some outdated and/or missing package. Google is your friend.

Let’s proceed with the compilation (takes a couple of minutes):


We run make qemu to start testing MentOS on a virtual machine.

To compile MentOS all subsequent times, you will only need to run the make command inside the build folder.

Congratulations, you’ve built an operating system!

Next steps

The next sections are for those who want to take a few more steps.

Changing Scheduling Algorithm

MentOS internally implements several scheduling algorithms and gives the user the ability to specify one in particular during the build process. To change scheduling algorithm, simply go to the build/ folder and reconfigure cmake by passing the -DSCHEDULER_TYPE flag with the name of the scheduling algorithm.


# Round Robin scheduling algorithm
# Priority scheduling algorithm
# Completely Fair Scheduling algorithm

Next, run make for compilation and make qemu for testing on virtual machine.


To open the debug interface, simply run the command make qemu-gdb inside the build folder. This will open three different windows: kernel booting to qemu, a shell with debug messages, and cgdb ready to stop/modify debugging.

Common troubleshooting

Warning on Cmake: CMP0037

  CMake Warning (dev) at programs/CMakeLists.txt:114 (add_custom_target):

  Policy CMP0037 is not set: Target names should not be reserved and should
  match a validity pattern.  Run "cmake --help-policy CMP0037" for policy
  details.  Use the cmake_policy command to set the policy and suppress this

  The target name "help" is reserved or not valid for certain CMake features,
  such as generator expressions, and may result in undefined behavior.

The commit 5363v115bf introduces the help command based on a program called help. When you run "cmake ..", all programs inside programs/ folder are added to CMake as commands (to be able to do make program_name later), but according to CMP0037 policy, some commands are reserved (like install, build), including help and that’s why the warning.

The warning can be ignored (although in more recent CMake releases it may be considered an error, aborting the compilation).

In Linux environments, there is no /bin/help but it is a built-in feature to the shell. A fix-warning-cmake is available here: fix-warning-cmake.diff. In order to apply the patch use git apply fix-warning-cmake.diff.

Unable to init server with QEMU

Unable to init server: Could not connect: Connection refused
qemu-system-i386: multiboot knows VBE. we don't

Most probably you have not installed/configured/enabled the graphical environment for qemu. This happens if you are on WSL (Windows subsystem for Linux) or if you are accessing the machine from text mode.

  1. On the other hand, the Mentos are candies: pay attention to the capitalized letters. To make Mentos you just need some sugar, glucose syrup, fully hydrogenated coconut oil, corn starch, natural mint flavors, maltodextrin and some thickeners: E418, E466, E414. ↩︎