The aspect that makes an image taken from a camera valued is that it is a represention of the real world.
The aspect that makes digitally created images valued is that is is a deliberate representation of an image in the artist’s mind.
Both of these are therefore valued because there is an element that makes it - in some sense - real.
Automatically generated images do not share this element. These images are not representations of the real world, nor are they a deliberate representation of an image in the artist’s mind. Generated images are loosely based on patterns from such representations - this is not sufficient.
Of course, these are not clear boundaries. And the appeal of automatically generated images cannot be ignored. It is an accelerant - imitating at a moments notice that which previously took much time and resources. It provides to the masses an ability they previously couldn’t achieve. However, these imitations mock that which we value in an image. If there are no clear boundaries, we must at least define some scale or varying degree.
Images taken from a camera require some signal processing and filtering to convert readings from a sensor to a digital image. This can be considered a minimum necessity. However, further modifications are done using software to change how the image appears - this is not particularly desirable (unless it is deliberate and explicit). Worse, some modern cameras use automatic generation to enchance the image!
Digital images are created using software. The artist has an image in mind, and the software is simply a medium through which it is expressed. Parts of this process can be delegated to automatic generation and classified by the control exerted:
How different is the result from the image that the artist had in mind?
The problem is that all these delegation types may eventually produce a comparable result. It is the artists responsibility to be deliberate and transparent, and it is the patrons responsibility to demand it.
What if the artist does not have a particular image in mind? It may be vague; it is not inconceivable that the process of creation influences and directs the final form. For such creations, the influence may be subject to the same scrutiny as the creation. Is the influence a true representation of the real world, or a deliberate representation of an artist?
Ultimately, this is a preference. It is possible to define the boundary concretely, to draw a line seperating that which is acceptable and that which is not - but this is subject to change and scrutiny. The artist must decide.
Summary:
Too little is not good.
Too much is not good.
These extremes are not great.
You must first realize this balance. If you listen carefully, too much and too little with guide you there.
And then you will realize your potential.
]]>
This method does not require an external installation media, but only works for local accounts. If the drive is encrypted with BitLocker, it’s key will be required (this method does not reset the key).
Source: https://dev.to/jackscott7/reset-windows-1011-password-without-usb-or-reinstall-43m6
Online Microsoft accounts require internet access to login; passwords are tied to this account and cannot be reset locally.
Some sources recommend pressing F11 on startup to enter “Recovery”, but this is not reliable; some machines may use F10 or F12 instead. Use the force shutdown method above instead.
Select “Troubleshoot”
Select “Advanced options”
Select “Command Prompt”
This will open a command prompt window.
Identify the operating system partition (value for “osdevice”). This is typically “C:\”, but it may be different.
bcdedit
Create a backup of “Utilman.exe”, and then replace it with “cmd.exe”:
copy C:\Windows\System32\Utilman.exe C:\Windows\System32\Utilman.bak
copy C:\Windows\System32\cmd.exe C:\Windows\System32\Utilman.exe
If prompted to overwrite, accept by typing “Yes”.
Windows is not case sensitive. If you experience difficulty due to keyboard layout issues, special symbols (like “\”) can be copied from the terminal prompt.
The command prompt window can be closed and the system can now be rebooted.
After startup, select the “Accessibility” icon at the bottom right of the screen. This will open a command prompt window.
List the users (to verify username) and change the password:
net user
net user "UserNameHere" "NewPasswordHere"
Although the next step is not necessary, it it recommended for security. Once logged in, open a command prompt and restore the Accessibility utility:
copy C:\Windows\System32\Utilman.bak C:\Windows\System32\Utilman.exe
If this returns access denied errors like above (even with admin privileges), boot back into recovery mode to restore.
]]>
No knowledge about the internal workings of the black box is expected. It may not be known, or you may not know it. The system is only known in terms of inputs and associated outputs.
All inputs are not the same. Some inputs control what the system does.
Other inputs control what the system operates on.
The output can be used to drive inputs - this is feedback.
To know the black box is to know the relation between it’s inputs and outputs; to contruct a model of how it might work.
A model may reveal the internal workings of the black box.
Does the black box make you uncomfortable? Does it leave you with the feeling that your knowledge is superficial or incomplete? Do you perhaps seek to know everything? If so, there is one more thing you should know.
All systems have black boxes. Complete knowledge of any system is not possible. Uncovering how a system works - and constructing more models - reveals more black boxes.
Some elements were inspired by https://en.wikipedia.org/wiki/Black_box.
]]>
Over billions of years, that simplicity becomes slightly more complex.
Complexity is accelerated by dependency.
That complexity can only persist because it gives enough time for the environment to support it - for it’s dependencies to persist. This is a delicate balance.
All complexities are not the same. The Earth has evolved to deal with much of it. However, some complexities - the ones we create - are very difficult to undo.
When that complexity far exceeds what the environment is capable of handling, it makes me sad.
Because that means one of two things:
This may not be inherently bad, but the future of life then becomes an uncertainty.
]]>
A terminal emulator provides an interface to the system using text commands. A core set of utilites are provided by default, and more can be installed and upgraded with pkg - the Termux package manager 2.
Refer to the official documentation to learn more about Termux: https://wiki.termux.dev/wiki/Main_Page.
The official source for installation is F-Droid, although builds are published on Github as well 3. Google PlayStore is not recommended. Do not mix applications between sources. Applications are installed by downloading and opening the Android Package Kit (APK) file.
To install applications from APK files, “Install from Unknown Sources” must be temporarily enabled in Settings for the file manager. For security, disable it once installation is completed.
The following applications need to be installed:
F-Droid does not need to be downloded; search for the “Download APK” link under the latest version.
termux is the main application. termux-api exposes device functionality like vibrate, clipboard, torch and fingerprint 4. termux-boot allows Termux to automatically start after device is restarted 5.
Remote acceses via Secure SHell (ssh).
Install and start sshd:
pkg install openssh
sshd
To use ssh, the hostname of the server and a username/password combination is required. Run whoami to determine the username (typically in the format u0_aXXX), and set the password with passwd.
The local IP address of the phone can be used to test the connection. The local IP address can be found under WiFi settings by expanding the currently connected network. Run ssh from a different machine on the same network:
ssh -p 8022 u0_aXXX@192.168.0.1
What follows is configuring key based authentication. While not strictly necessary, it is recommended for security. It can be configured later if required.
The password will be prompted when connecting. This is not very secure since attackers could try and guess the username/password. To increase security, switch to key-based authentication by generating a key with ssh-keygen on the machine you will be connecting to the phone from:
ssh-keygen -t rsa
Enter file in which to save the key (/home/username/.ssh/id_rsa): /home/username/.ssh/keyname
Enter passphrase for "key-filename" (empty for no passphrase):
This will create two files under /home/username/.ssh:
keyname: a passphrase protected private keykeyname.pub: a public keyThe public key’s content must be copied to the file /data/data/com.termux/files/home/.ssh/known_hosts. This can be done using USB, by copying the text to clipboard and pasting it over ssh, or by using ssh-copy-id:
ssh-copy-id -p 8022 -i /home/username/.ssh/keyname.pub u0_aXXX@192.168.0.1
The private key can now be used to connect to the phone without a password:
ssh -p 8022 -i /home/username/.ssh/keyname u0_aXXX@192.168.0.1
Once the key-based authentication has been tested, disable password authentication by editing /data/data/com.termux/files/usr/etc/ssh/sshd_config and setting PasswordAuthentication no. Restart sshd for the changes to take effect:
pkill sshd
sshd
Try connecting from a different machine without the key; this should now fail:
ssh -o PasswordAuthentication=yes \
-o PreferredAuthentications=keyboard-interactive,password \
-o PubkeyAuthentication=no \
u0_aXXX@192.168.0.1
Keep the private key at /home/username/.ssh/keyname secure - you cannot login remotely without it if “PasswordAuthentication” is disabled.
Web server to host a website.
Install httpd:
pkg install apache2
Edit /data/data/com.termux/files/usr/etc/apache2/httpd.conf:
# Set server hostname and path to executables
ServerName 127.0.0.1
ServerRoot "/data/data/com.termux/files/usr"
# Set port which server accepts connections over
Listen 8080
# Set directory from which Apache will serve files
DocumentRoot "/data/data/com.termux/files/home/folderToServe"
# Set the file to save errors to
ErrorLog "/data/data/com.termux/files/home/httpd.log"
# Handle concurrent connections with Worker Multi-Processing Module
LoadModule mpm_worker_module libexec/apache2/mod_mpm_worker.so
# Maps file extensions to content types
LoadModule mime_module libexec/apache2/mod_mime.so
# Ensures proper privilege handling on Unix systems
LoadModule unixd_module libexec/apache2/mod_unixd.so
# Set default webpage to return when client connects to the server
LoadModule dir_module libexec/apache2/mod_dir.so
DirectoryIndex index.html
# Prevent arbitrary file requests (lock down filesystem)
LoadModule authz_core_module libexec/apache2/mod_authz_core.so
<Directory "/">
Require all denied
</Directory>
# Allow access only from local connections (127.0.0.1)
LoadModule authz_host_module libexec/apache2/mod_authz_host.so
<Directory "/data/data/com.termux/files/home/folderToServe">
Require local
</Directory>
Replace folderToServer with the path your website is saved at. Modules are available for more functionality - refer to the Apache http server documentation: https://httpd.apache.org/docs/current/mod/.
A folder from a different machine can be copied to the phone over ssh using rsync (without the --dry-run flag):
rsync -havP --checksum --delete --dry-run \
-e 'ssh -i /home/username/.ssh/keyname'
/local/path/to/folder/in/machine \
u0_aXXX@192.168.0.1:/data/data/com.termux/files/home/folderToServe
Test to ensure the server starts without errors by running httpd. If the command doesn’t fail due to invalid configuration, errors will be logged to the file specified above by ErrorLog. The files under DocumentRoot should now be served - visit 127.0.0.1:8080 on the phone’s browser.
The proxy to the Internet.
To access a server over the internet, the server needs a public IP address. A Virtual Private Server (VPS) is a publicly accessible machine hosted by a cloud provider. Since the phone itself does not have a public IP address, it must use a reverse proxy like frp or nginx running on a VPS to be accessible on the internet. An alternative is to use the Tor network.
The Tor network is a group of volunteer-operated servers 6. It uses a special protocol for privacy and anonymity. A Tor service can be set up to make the local services accessible to the internet.
A Tor service address is different from public domains; this is the link for the Tor Project homepage:
http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/
It can be accessed using the Tor Browser, and will not work with browsers that do not route traffic through the Tor network.
To setup the Tor service, edit tor run config /data/data/com.termux/files/usr/etc/tor/torrc and append:
# Service for sshd
HiddenServiceDir /data/data/com.termux/files/home/.tor/ssh/
HiddenServicePort 22 127.0.0.1:8022
# Service for httpd
HiddenServiceDir /data/data/com.termux/files/home/.tor/webserver/
HiddenServicePort 80 127.0.0.1:8080
HiddenServiceDir defines where the identity files for the service must be stored. HiddenServicePort maps a virtual port to a local port. Clients will target the virtual port (22 or 80) and tor will forward this traffic to the specified local port (8022 or 8080).
Execute tor and get the hostame of the service:
tor
cat /data/data/com.termux/files/home/.tor/ssh/hostname
Use torsocks to proxy ssh connections:
torsocks ssh -p 22 -i /home/username/.ssh/keyname.pub \
u0_aXXX@abcde23456abcde23456abcde23456abcde23456abcde23456abcdef.onion
The Termux documentation recommends
proxychains-ngovertorsocksastorsocksmay not work since Android 8 7.
Use the Tor Browser to visit the website. The services hosted on Android with Termux can now be accessed from anywhere in the world with an internet connection and Tor.
Keeping the services alive.
When the phone is rebooted, the services will stop. To automatically start up, the service must be enabled and Termux must be started on boot.
Services run in the background and restart when certain conditions are triggered.
Install the service manager:
pkg install termux-services
Start the services and enable it to autostart when Termux starts.
sv up sshd
sv up httpd
sv up tor
sv-enable sshd
sv-enable httpd
sv-enable tor
The number of services supported is limited; refer to the documentation to learn more: https://wiki.termux.dev/wiki/Termux-services.
The logs for running services are stored at /data/data/com.termux/files/usr/var/log/sv/.
To ensure that the serivce works, restart the service after killing all currently running processes:
sv down httpd
pgrep httpd
pkill -f httpd
sv up httpd
cat /data/data/com.termux/files/usr/var/log/sv/httpd/current
Create the directory for boot scripts:
mkdir -p /data/data/com.termux/files/home/.termux/boot
Create a file (nvim start-services) under the boot directory:
#!/data/data/com.termux/files/usr/bin/sh
termux-wake-lock
. $PREFIX/etc/profile
termux-wake-lock stops Android from going into deep sleep - this prevents Android’s aggressive power management from suspending background processes. The second line sources /data/data/com.termux/files/usr/etc/profile - this configures the environment.
It matters because we wish to be happy. To be happy as long as we can. To never be unhappy - if at all possible. Is it possible?
That largely depends on our the measure of happiness.
Happiness could be:
What is happiness for you?
How would you measure it?
Rhythmbox-Reloaded is a C++ application that needs to play audio files. There are libraries that handle this - they simply need to be included, initialized and used. miniaudio is easy to integrate, but has file format limitations. libvlc can support almost any format, but bundling it with the application impacts size and performance. Dynamic linking addresses this issue, but creates a requirement for external dependencies. As a compromise, the application may try to use libvlc, but fall back to miniaudio.
Miniaudio is an audio playback library for C and C++; it is made up of a single source file without external dependencies.
To use the library, download the header file miniaudio.h.
To test the library by playing a file, create ma-test.cpp:
#define MINIAUDIO_IMPLEMENTATION
#include "miniaudio.h"
#include <iostream>
int main(){
// Miniaudio variables
ma_engine engine;
ma_result result;
ma_sound sound;
// Initialize engine
result = ma_engine_init(NULL, &engine);
if (result == MA_SUCCESS){
// Load file
result = ma_sound_init_from_file(&engine, "file.mp3", 0, NULL, NULL, &sound);
if (result == MA_SUCCESS){
// Start playback
ma_sound_start(&sound);
std::cout << "Playing file.\n";
// Wait until completion
while(!ma_sound_at_end(&sound)){
ma_sleep(10);
}
ma_sound_uninit(&sound);
std::cout << "Playback completed.\n";
} else {
std::cout << "Failed to load file.\n";
}
} else {
std::cout << "Failed to initialize file.\n";
}
}
Compile and run:
g++ ma-test.cpp -o ma-test.o
./ma-test.o
Ensure the header file
miniaudio.hand audio filefile.mp3are in the same folder as the above source code file.
The high level API has built in support for WAV, MP3 and FLAC - other formats require more complex configuration with the low level API. 1 If support for an extensive list of audio formats is required, libvlc may be better suited.
libvlc is a C library that is the core engine of the VLC media player. VideoLAN provides official C++ bindings (libvlcpp) that acts as a wrapper for libvlc. The bindings consist of header files; only the vlcpp folder - containing several .hpp files - is required from the repository: https://code.videolan.org/videolan/libvlcpp.
VLC and libvlc are required dependencies. Some sources of VLC may not expose libvlc as a library. Refer to the VLC installation guides for your particular system.
Once libvlc is accessible and the vlcpp folder has been copied to the current folder, the engine can be tested; create vlc-test.cpp:
#include "vlcpp/vlc.hpp"
#include <unistd.h>
int main(){
// VLC variables
VLC::Instance m_instance;
VLC::MediaPlayer m_player;
VLC::Media m_media;
// Initialize with video disabled
static const char* const vlc_args[] {"--no-video", "--vout=dummy"};
m_instance = VLC::Instance(int(sizeof(vlc_args)/sizeof(vlc_args[0])), vlc_args);
m_player = VLC::MediaPlayer(m_instance);
m_media = VLC::Media(m_instance, "file.mp3", VLC::Media::FromPath);
m_player.setMedia(m_media);
// Start playback
if (m_player) {
m_player.play();
// Wait for playback to start, then finish
while (!m_player.isPlaying()){ usleep(10); }
while (m_player.isPlaying()) { usleep(10); }
}
return 0;
}
libvlc must be linked when compiling:
g++ -lvlc vlc-test.cpp -o vlc-test.o
./vlc-test.o
If you need to package the application as an AppImage, you must exclude libvlc when building. VLC initialization unexpectedly fails with partial vlc installations (missing some plugins) and including the full installation would be quite inefficient - it is better to dynamically link.
It would be beneficial if the application could retain some functionality even if libvlc is unavailable. When using libvlc by #include <vlcpp/vlc.hpp>, the application does not start without libvlc.
Using preprocessor directives only works for compilation. The developer machine can be required to have vlc installed, but this doesn’t solve the runtime issue.
Another option is to load the library at runtime using dlopen or LoadLibrary, but since libvlcpp is being used as a binding, this doesn’t seem easily possible.
Currently, fallback is not supported. Two build version may be offered - one using miniaudio and the other libvlc. An integrated version would be prefered.
The requirement is to implement a new database design for managing a variety of digital assets. Each asset has an associated identifier (hash) and metadata. The hash is calculated using a tool like sha512sum or phash. Metadata is extracted in JSON format using tools like ffprobe or exiftool. Ideally, only specific required metadata fields will be stored in the database, but that makes future changes more complex and requires making certain assumptions about the type of digital asset. To make the database as general purpose as possible, the database management system (DBMS) needs to support JSON documents.
PostgreSQL is an open source relational database that supports JSON types. To better understand the configuration of this database, it is beneficial to briefly cover some terminology and hierarchies.
PostgreSQL uses a client-server model. The server accepts client connections at a specified port (5432 by default) and stores data at a specified location (typically /usr/local/pgsql/data or /var/lib/postgresql/data). The server can manage any number of databases stored in this data directory; this forms a single database cluster. The database itself consists of schemas; schemas act as a namespace for organizing database objects like tables and functions (SQL queries).
PostgreSQL uses roles and privileges for access control. Roles are created and assigned permissions to perform specific actions:
create role role_name password 'role_password';
grant usage on schema schema_name to role_name; # allows accessing objects under specified schema
grant INSERT on schema.table to role_name; # allows action on specified object
The specified action can be ALL, or INSERT, SELECT, UPDATE, DELETE and so on.
Docker is used to deploy the services. Three services are used:
db: the PostgreSQL server and database clusteradminer: webpage that allows running SQL commandsapi: PostgREST web server that exposes a REST API
psql is a terminal based client interface for the PostgreSQL server. pgAdmin and phpMyAdmin are graphical interfaces similar to adminer; GUIs make running SQL commands easier by eliminating the need for terminal access. Adminer can be used to create schemas, perform CRUD operations on tables, create roles, assign permissions and execute other SQL commands.
The api is used to enable clients to perform actions without specialized tools for connecting to the database; only the ability to send HTTP requests is required.
Create docker-compose.yml:
services:
db:
image: postgres:18.2
restart: always
environment:
POSTGRES_USER: ${ADMIN_USERNAME}
POSTGRES_PASSWORD: ${ADMIN_PASSWORD}
POSTGRES_DB: ${DATABASE_NAME}
volumes:
- pgdata:/var/lib/postgresql
adminer:
image: adminer:5.4.2
restart: always
ports:
- 8080:8080
depends_on:
- db
api:
image: postgrest/postgrest:v14.5
ports:
- 3000:3000
environment:
PGRST_SERVER_HOST: 0.0.0.0
PGRST_DB_URI: postgres://${AUTH_ROLE_USERNAME}:${AUTH_ROLE_PASSWORD}@db:5432/${DATABASE_NAME}
PGRST_OPENAPI_SERVER_PROXY_URI: http://127.0.0.1:3000
PGRST_JWT_SECRET: ${JWT_SECRET}
depends_on:
- db
volumes:
pgdata:
Define the variables in .env:
ADMIN_USERNAME=
ADMIN_PASSWORD=
DATABASE_NAME=
DATABASE_SCHEMAS=
AUTH_ROLE_USERNAME=
AUTH_ROLE_PASSWORD=
JWT_SECRET=
Use
echo $(< /dev/urandom tr -dc A-Za-z0-9 | head -c32)to generate secure alphanumeric passwords; change length (to-c64,c128, …) as required. The JWT secret must be 32 characters.
Start the services with docker compose up -d.
Two roles are used to separate authentication (verifying identity) from authorization (verifying permissions). Clients connect to the authenticator role by providing the correct password, but this role does not have any permissions. The client can then switch to another role (SET ROLE role_name) which has the required permissions.
The following can be executed using a client interface (like psql or Adminer):
create role role_name nologin;
create role authenticator noinherit login password 'example_authenticator_password';
grant role_name to authenticator;
create schema schema_name;
grant usage on schema schema_name to role_name;
grant all on schema_name.table_name to role_name;
The nologin rolename and authenticator rolename and password should match that defined in
.env.
nologin disables connecting directly as the specified rolenoinherit prevents default permissions from being automatically assignedThe containers need to be restarted for postgrest to start without errors:
docker compose down
docker compose up -d
Check for keywords error or fatal in the logs (docker compose logs).
The following steps are from the PostgREST documentation 1.
Use the JWT secret defined in .env. Create a signed token using bash and openssl, replacing role_name with the nologin role:
#!/bin/bash
set -e
JWT_SECRET='test_secret_that_is_at_least_32_characters_long'
_base64 () { openssl base64 -e -A | tr '+/' '-_' | tr -d '='; }
header=$(echo -n '{"alg":"HS256","typ":"JWT"}' | _base64)
payload=$(echo -n "{\"role\":\"role_name\"}" | _base64)
signature=$(echo -n "$header.$payload" | openssl dgst -sha256 -hmac "$JWT_SECRET" -binary | _base64)
echo -n "$header.$payload.$signature"
Run this script (./script_name.sh) to create the token.
Set your token value (this needs to be repeated when a new shell is opened):
read TOKEN
To retrieve records:
curl https://domain_name.org/table_name -X GET \
-H "Authorization: Bearer $TOKEN" \
-H "Accept-Profile: schema_name"
Note, the schema must be created manually (using Adminer) and defined in the environment (
DATABASE_SCHEMAS) with the containers restarted after updates for the schema to be accessible. 2
To insert new data:
curl https://domain_name.org/table_name -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "{"column1": "value", "column2": {key1: value1, key2: value2}, "column3": 3 }"
There are two different ways of approaching troubleshooting - by starting with something that doesn’t work and removing/modifying components until does, or by starting with something that does work and adding components until it doesn’t. The second approach is easier when there are many things that could go wrong.
A common error that may be faced is that the binding port is already used. The syntax for port binding is - host_port:container_port. The host post may need to be changed if there is a conflict.
When modifying the docker-compose.yml, if you mount the pgdata volume to /var/lib/postgresql/data and try to access the API, it will state that the JWT secret is not set. If you have a look at logs of the PostgreSQL service, you will notice that it is actually not running - the error references a pull request which requires that the volume be mounted at /var/lib/postgresql instead for PostgreSQL v18+ 3. Determining this true cause by troubleshooting the API would have been tedious and time consuming.
The PostgREST REST API requires cache invalidation; if faced with errors of the format Could not find xxx in the schema cache and after verification that xxx indeed exists in the specified schema, simply stop and start the service to update cache.
Note, one PostgREST instance can only connect to one database instance. If API access is required for multiple databases in the same database cluster, each database requires a PostgREST instance 4. Depending on your requirements, using multiple schemas may be a better option.
]]>The first step is to get an executable for the developer machine. This is quite simple - it can be done in the Qt Creator IDE or using the terminal with qmake and make. qmake generates a Makefile based on the contents of a project file.
The specifics of the project file aren’t important; this file is created automatically by QtCreator and defines a few configurations (compiler versions and required Qt modules) as well as files to include and compile. Here is a minimal example of a project file Application.pro:
QT += core gui
greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
CONFIG += c++17
SOURCES += \
application-src/main.cpp \
application-src/mainwindow.cpp
HEADERS += \
application-headers/mainwindow.h
FORMS += \
application-forms/mainwindow.ui
INCLUDEPATH += application-headers/
INCLUDEPATH += application-src/
RESOURCES += \
application-styling/application-styling.qrc
qmake reads this file and generates a Makefile:
#############################################################################
# Generated by qmake (3.1) (Qt 6.2.4)
#############################################################################
####### Compiler, tools and options
CC = gcc
CXX = g++
MKDIR = mkdir -p
COPY = cp -f
####### Output directory
OBJECTS_DIR = ./
####### Files
SOURCES = ../application-src/main.cpp \
../application-src/mainwindow.cpp
OBJECTS = main.o \
mainwindow.o
DIST = /usr/lib/x86_64-linux-gnu/qt6/mkspecs/features/spec_pre.prf \
/usr/lib/x86_64-linux-gnu/qt6/mkspecs/common/unix.conf \
/usr/lib/x86_64-linux-gnu/qt6/mkspecs/common/linux.conf \
../application-src/main.cpp \
../application-src/mainwindow.cpp
####### Build rules
Application: ui_mainwindow.h $(OBJECTS)
$(LINK) $(LFLAGS) -o $(TARGET) $(OBJECTS) $(OBJCOMP) $(LIBS)
####### Sub-libraries
mocclean: compiler_moc_header_clean compiler_moc_objc_header_clean compiler_moc_source_clean
####### Compile
main.o: ../application-src/main.cpp ../application-headers/mainwindow.h
$(CXX) -c $(CXXFLAGS) $(INCPATH) -o main.o ../application-src/main.cpp
####### Install
install_target: first FORCE
@test -d $(INSTALL_ROOT)/opt/Application/bin || mkdir -p $(INSTALL_ROOT)/opt/Application/bin
$(QINSTALL_PROGRAM) $(QMAKE_TARGET) $(INSTALL_ROOT)/opt/Application/bin/$(QMAKE_TARGET)
-$(STRIP) $(INSTALL_ROOT)/opt/Application/bin/$(QMAKE_TARGET)
Running make uses the GNU Compiler Collection to generate an elf binary that can be executed.
A critical limitation of the executable generated above is that it depends on certain libraries in a way that makes it distribution dependent.
The most likely way this issue would be discovered is by running the application on different distributions - typically a given with even with a small user base. However, for this particular project, the issue was intrestingly identified in the CI/CD pipeline.
You may wish to automate the building and publishing of the application when the source code is pushed to remote git repository. Git hosting services often include an “Actions” feature that can use docker to achieve this. A minimal image that is often chosen is Alpine Linux. Required packages are installed with the Alpine package manager apk and the process for building the application outlined above is followed.
To test the generated binary, run it on the development machine. In this case, it happens to be Arch Linux - an operating system that is organised differently to Alpine Linux. Running the binary results in an error: Error: file not found.
To understand the cause of this error, a better understanding is required of how Make and the GNU compiler works.
The compiler’s actions can be divided into 4 stages handled by different components.
| Component | Purpose |
|---|---|
| Preprocessor | expand or resolve lines starting with the directive (’#') |
| Compiler | translates source code to assembly code |
| Assembler | translates assembly code to object code |
| Linker | combines multiple object files and links external libraries |
Examining the output of each stage gives a better insight into the process.
The component of importance here is the linker.
Dynamic link object files are identified by the .so for “shared object” extension on Linux and .dll for “dynamically linked library” on Windows.
When linking, the symbols of the object file is added instead of the entire file itself.
The following command lists all the symbols in the libcurl library:
objdump -T /usr/lib/libcurl.so
During runtime, missing symbols are resolved by the system. This makes the application more resource efficient and means changes to shared objects can be easily propagated. The comes at the cost of comptatibility since more assumptions are made about the system - such as having the shared object in an accessible path.
All object files are stored directly in the resulting executable. Statically linked libraries are identified by an .a for “archive” extension.
Static linking results in better compatibility since it minimizes external dependencies. However, is increases resource consumption and makes it more difficult to update.
Static linking Qt is not trivial. The official installation only includes dynamic libraries and static linking may potentially face licensing restrictions. Static linking requires building Qt from source and additional configuration.
AppImages takes a different approach. The executable binary is generated using the same process outlined before, but this is then packaged with most required files to run with minimal external dependencies.
It should be noted that AppImages do still have some dependencies, but these are likely fulfilled by default by your distribution. There are some exceptions; Filesystem in Userspace (FUSE), which allows non-root users to mount filesystems and is required for the AppImage to work, is not bundled with older AppImages and may need to be installed explicitly.
If this is your first time packaging an application with AppImage, getting it to work may take time. After much trial and error, it seems that packaging with AppImage has two requirements: a compliant AppDir and a specific version of glibc.
Creating the AppDir is quite simple and can be done manually; this is how it is organized:
.
└── AppDir
└── usr
├── bin
│ └── app-name.elf
└── share
├── applications
│ └── app-name.desktop
└── icons
└── hicolor
└── 256x256
└── apps
└── app-name.svg
app-name.svg is an image file that will be displayed as the application’s icon. app-name.elf is the binary that was generated.
app-name.desktop is a desktop entry file:
Type=Application
Exec=app-name
Name=app-name
Icon=app-name
Categories=Utility;
Note that the file extension is excluded in the entries for
ExecandIcon.
glibc is a standard C library that provides many common and important functions. The glibc contraint is a little more nuanced.
To produce binaries that are compatible with many target systems, build on the oldest still-supported build system.
linuxdeployqt - the chosen packaging tool - recommends Ubuntu and enforces this system constraint quite strictly to encourage developers to support the most platforms.
The initial attempt on Alpine Linux failed because the glibc version was too new. From the documentation, it wasn’t entirely clear if “still-supported” referred to the “end of standard support” or “end of life”. Ubuntu 14.04 was the oldest system that had not reached end of life. Not only did the system have network issues (there was a bug with libcurl3 that made TLS connections unusable), but the application didn’t even compile. Sections of the source code needed to be modified because the glibc version was too old and didn’t recognise some of the newer syntax.
At this stage, the packaging tool was abandoned and alternatives were tested - this time on Arch Linux, a rolling distribution with the latest versions of all packages. After much frustration, the issue was narrowed down to the glibc version again (this was not as obvious as it is in hindsight). Arch Linux does not support rolling back to a specific version, so the correct version was manually downloaded from https://archive.archlinux.org/packages/g/glibc/ and forcefully installed. This promptly broke the system, causing a blue screen of death on shutdown (a first time on Linux). Luckily this was inside a virtual machine. Before shutdown, new terminals could not be opened, but some commands in an already open terminal worked and was used to copy some important data to a thumbdrive. The thumbdrive later failed due to an unrelated issue.
Ubuntu 22.04 - the oldest system that still had standard support - was used next. Everything went smoothly once the format for the AppDir was established and application was successfully packaged and run on different distributions.
A example of a command that might be used to create an AppImage on Ubuntu 22.04:
./linuxdeploy.AppImage -appimage \
AppDir/usr/share/applications/app-name.desktop \
-qmake=/usr/bin/qmake6 qmldir=application-styling/ \
-qmldir-importscanner=/usr/lib/qt6/libexec/qmlimportscanner
The
qmldirandimportscannerare required to include styling and image assets defined as Qt Resources.
It is intresting to see the lifespan of the software we build, and the chain of dependencies it requires.
]]>
Create and setup the virtual environment:
python3 -m virtualenv env-name
source env-name/bin/activate
pip3 install jupyter
Run Jupyter Notebook:
jupyter notebook
While it is very easy to install modules by sourcing the correct virtualenv and using pip, there are a few drawbacks to this approach. Jupyter Notebook needs to be installed for each new environment created. If used as a server, this would become quite tedious and difficult to maintain - particularly if there is a need to use multiple environemnts at once with different users.
The Jupyter Notebook installation can be centralized using the system package manager. Virtual environments are created separately.
To install Jupyter Notebook on Debian based systems, use apt:
sudo apt install python3 python3-virtualenv jupyter
jupyter notebook
This is convenient because there is only one Jupyter Notebook installation - it can be accessed on localhost:8888. Next, virtual environments must created, and modules need to be installed in different virtual environments.
Commands can be run from within a notebook. A better idea of how Jupyter Notebook works is needed to understand this. Internally, it has a component called the IPython Kernel. This is responsible for communicating with the Python interpreter on the system. The ipykernel is included by default when Jupyter Notebook is installed, but it can also be installed independently.
Shell commands are provided by the operating system. They can be run by prefixing a line with an exclaimation mark:
!ls ; cd example_dir
This spawns a subprocess that exits upon completion; it is equivalent (on Linux) to:
import subprocess
subprocess.run(["bash", "-c", "ls ; cd example_dir"])
In this case, the ls will list the files in the current directory and cd will change the directory. The change in directory will not persist since the process exists after completion.
Magic commands (prefixed with a percentage sign) are provided by the IPython kernel. State changes persist because these commands are run inside the kernel process itself (and apply to the virtual environment in which the kernel was installed):
%cd example_dir
%page my_file.txt
%pip install bs4
This prints example_dir/my_file.txt and installs BeautifulSoup (a module for HTML parsing) in the current virtual environment.
For all available magic commands, refer to the IPython documentation or run:
%lsmagic
To install a module in the current virtual environment:
%pip install module_name
It is not possible to create and use a virtual environment directly within a notebook. For example:
!python -m virtualenv myvenv
!source myvenv/bin/activate
!pip install bs4
The first two commands succeed but the third fails with an externally-managed-environment error. This is because a new subprocess is created for each command - the sourced environment is not actually activated when pip is used. This can be resolved by chaining the command:
!python -m virtualenv myvenv; source myvenv/bin/activate; pip install bs4
BeautifulSoup now installs in the myvenv virtual environment, but the notebook does not use this environment. The virtual environment that the IPython kernel is running on is different to the one created - virtual environments cannot be nested!
To make a virtual environments available to Jupyter Notebook, the ipykernel module should be installed inside the virtual environment:
python3 -m virtualenv --system-site-packages env-name
source env-name/bin/activate
pip install ipykernel
python3 -m ipykernel install --user --name=env-name --display-name "Python (env-name)"
The
--system-site-packagesflag is particularly important if you depend on globally installed modules. For example,matplotlibhas to be specially packaged for ARM platforms like Raspberry Pi and installed using the system package manager. This flag ensures that these modules can be found.
Since the Jupyter Notebook installation is global, the above commands can be easily run from the interface by creating a new terminal.
When creating a new notebook, you can select Python (env-name) to use a particular virtual environment.
To list all kernels and remove a particular kernel:
jupyter-kernelspec list
jupyter-kernelspec uninstall env-name
There is still a problem - this is running directly on the server. If something goes wrong, it might affect other services. This is also not very easily portable.
Using docker, this can be containerized!
This Dockerfile uses Alpine linux as the base image and installs python, pip, virtualenv and jupyter-notebook using the Alpine package manager apk.
The last command starts the notebook. This probably isn’t very secure - you might want to modify the flags based on your security model.
FROM alpine:3.22
RUN apk add python3 py3-pip py3-virtualenv jupyter-notebook
EXPOSE 8888
CMD ["jupyter-notebook", "--ip=0.0.0.0", "--no-browser", "--allow-root", "--NotebookApp.token='your-very-long-and-secure-passphrase'"]
Now, creating the following docker-compose.yaml file and running docker compose up will build the image and expose Jupyter Notebook on localhost:4444. To manage the virtual environments, use the terminal and follow the steps in the previous section.
services:
jupyter:
container_name: jupyter
build: .
ports:
- 4444:8888
To reproduce this cionfiguration on a different service, just the Dockerfile and docker-compose.yaml need to be copied.