Latest posts for tag debian

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

Building Qt5 takes a long time. The build server I was using had CPUs and RAM, but was very slow on I/O. I was very frustrated by that, and I started evaluating alternatives. I ended up setting up scripts to automatically provision a throwaway cloud server at Hetzner.

Initial setup

I got an API key from my customer's Hetzner account.

I installed hcloud-cli, currently only in testing and unstable:

apt install hcloud-cli

Then I configured hcloud with the API key:

hcloud context create

Spin up

I wrote a quick and dirty script to spin up a new machine, which grew a bit with little tweaks:

#!/bin/sh

# Create the server
hcloud server create --name buildqt --ssh-key … --start-after-create \
                     --type cpx51 --image debian-10 --datacenter …

# Query server IP
IP="$(hcloud server describe buildqt -o json | jq -r .public_net.ipv4.ip)"

# Update ansible host file
echo "buildqt ansible_user=root ansible_host=$IP" > hosts

# Remove old host key
ssh-keygen -f ~/.ssh/known_hosts -R "$IP"

# Update login script
echo "#!/bin/sh" > login
echo "ssh root@$IP" >> login
chmod 0755 login

I picked a datacenter in the same location as where we have other servers, to get quicker data transfers.

I like that CLI tools have JSON output that I can cleanly pick at with jq. Sadly, my ISP doesn't do IPv6 yet.

Since the server just got regenerated, I remove a possibly cached host key.

Provisioning the machine

One git server I need is behind HTTP authentication. Here's a quick hack to pass the relevant .netrc credentials to ansible before provisioning:

#!/usr/bin/python3

import subprocess
import netrc
import tempfile
import json

login, account, password = netrc.netrc().authenticators("…")

with tempfile.NamedTemporaryFile(mode="wt", suffix=".json") as fd:
    json.dump({
        "repo_user": login,
        "repo_password": password,
    }, fd)
    fd.flush()

    subprocess.run([
        "ansible-playbook",
        "-i", "hosts",
        "-l", "buildqt",
        "--extra-vars", f"@{fd.name}",
        "provision.yml",
        ], check=True)

And here's the ansible playbook:

#!/usr/bin/env ansible-playbook

- name: Install and configure buildqt
  hosts: all
  tasks:
   - name: Update apt cache
     apt:
        update_cache: yes
        cache_valid_time: 86400

   - name: Create build user
     user:
        name: build
        comment: QT5 Build User
        shell: /bin/bash

   - name: Create sources directory
     become: yes
     become_user: build
     file:
        path: ~/sources
        state: directory
        mode: 0755

   - name: Download sources
     become: yes
     become_user: build
     get_url:
        url: "https://…/{{item}}"
        dest: "~/sources/{{item}}"
        mode: 0644
     with_items:
      - "qt-everywhere-src-5.15.1.tar.xz"
      - "qt-creator-enterprise-src-4.13.2.tar.gz"

   - name: Populate home directory
     become: yes
     become_user: build
     copy:
        src: build
        dest: ~/
        mode: preserve

   - name: Write .netrc
     become: yes
     become_user: build
     copy:
        dest: ~/.netrc
        mode: 0600
        content: |
           machine …
           login {{repo_user}}
           password {{repo_password}}

   - name: Write .screenrc
     become: yes
     become_user: build
     copy:
        dest: ~/.screenrc
        mode: 0644
        content: |
           hardstatus alwayslastline
           hardstatus string '%{= cw}%-Lw%{= KW}%50>%n%f* %t%{= cw}%+Lw%< %{= kK}%-=%D %Y-%m-%d %c%{-}'
           startup_message off
           defutf8 on
           defscrollback 10240

   - name: Install base packages
     apt:
        name: git,mc,ncdu,neovim,eatmydata,devscripts,equivs,screen
        state: present

   - name: Clone git repo
     become: yes
     become_user: build
     git:
        repo: https://…@…/….git
        dest: ~/…

   - name: Copy Qt license
     become: yes
     become_user: build
     copy:
        src: qt-license.txt
        dest: ~/.qt-license
        mode: 0600

Now everything is ready for a 16 core, 32Gb ram build on SSD storage.

Tear down

When done:

#!/bin/sh
hcloud server delete buildqt

The whole spin up plus provisioning takes around a minute, so I can do it when I start a work day, and take it down at the end. The build machine wasn't that expensive to begin with, and this way it will even be billed by the hour.

A first try on a CPX51 machine has just built the full Qt5 Everywhere Enterprise including QtWebEngine and all its frills, for amd64, in under 1 hour and 40 minutes.

Here are the slides of mine and Ulrike's talk Doing things /together/.

Our thoughts about cooperation aspects of doing things together.

Sometimes in Debian we do work together with others, and sometimes we are a number of people who work alone, and happen to all upload their work in the same place.

In times when we have needed to take important decisions together, this distinction has become crucial, and some of us might have found that we were not as good at cooperation as we would have thought.

This talk is intended for everyone who is part of a larger community. We will show concepts and tools that we think could help understand and shape cooperation.

Video of the talk:

The slides have extensive notes: you can use ViewNotes in LibreOffice Impress to see them.

Here are the Inkscape sources for the graphs:

Here are links to resources quoted in the talk:

In the Q&A, pollo asked:

How can we still have a good code review process without making it a "you need to be perfect" scenario? I often find picky code reviews help me write better code.

Ulrike wrote a more detailed answer: Code reviews: from nitpicking to cooperation

Snap guides

Dragging from the rulers does not always create snap guides. If it doesn't, click on the slide background, "Snap guides", "Insert snap guide". In my case, after the first snap guide was manually inserted, it was possible to drag new one from the rulers.

Master slides

How to edit a master slide

  • Show master slides side pane
  • Right click on master slide
  • Edit Master...
  • An icon appears in the toolbar: "Close Master View"
  • Apply to all slides might not apply to the first slide created as the document was opened

Change styles in master slide

Do not change properties of text by selecting placeholder text in the Master View. Instead, open the Styles and formatting sidebar, and edit the styles in there.

This means the style changes are applied to pages in all layouts, not just the "Title, Content" layout that is the only one editable in the "Master View".

How to duplicate a master slide

There seems to be no feature implemented for this, but you can do it, if you insist:

  • Save a copy of the document
  • Rename the master slide
  • Drag a slide, that uses the renamed master slide, from the copy of the document to the original one

It's needed enough that someone made a wikihow: https://www.wikihow.com/Copy-a-LibreOffice-Impress-Master-Slide.

How to change the master slide for a layout that is not "Title, Content"

I could not find a way to do it, but read on for a workaround.

I found an ask.libreoffice.org question that went unanswered.

I asked on #libreoffice on IRC and got no answer:

Hello. I'm doing the layout for a presentation in impress, and I can edit all sorts of aspects of the master slide. It seems that I can only edit the "Title, Content" layout of the master slide, though. I'd like to edit, for example, the "Title only" layout so that the title appears in a different place than the top of the page. Is it possible to edit specific layouts in a master page?

In the master slide editor it seems impossible to select a layout, for example.

Alternatively I tried creating multiple master slides, but then if I want to create a master slide for a title page, there's no way to remove the outline box, or the title box.

My work around has been to create multiple master slides, one for each layout. For a title layout, I moved the outline box into a corner, and one has to remove it manually after create a new slide.

There seems to be no way of changing the position of elements not found in the "Title, Content" layout, like "Subtitle". On the other hand, given that one's working with an entirely different master slide, one can abuse the outline box as a subtitle.

Note that if you later decide to change a style element for all the slides, you'll need to go propagate the change to the "Styles and Formatting" menu of all master slides you're using.

From https://en.wikipedia.org/wiki/Dragonfly#Sex_ratios:

Sex ratios

The sex ratio of male to female dragonflies varies both temporally and spatially. Adult dragonflies have a high male-biased ratio at breeding habitats. The male-bias ratio has contributed partially to the females using different habitats to avoid male harassment.

As seen in Hine's emerald dragonfly (Somatochlora hineana), male populations use wetland habitats, while females use dry meadows and marginal breeding habitats, only migrating to the wetlands to lay their eggs or to find mating partners.

Unwanted mating is energetically costly for females because it affects the amount of time that they are able to spend foraging.

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

After having had some success with a sysroot in having a Qt5 cross-build environment that includes QtWebEngine, the next step is packaging the sysroot so it can be available both to build the cross-build environment, and to do cross-development with it.

The result is this Debian source package which takes a Raspberry Pi OS disk image, provisions it in-place, extracts its contents, and packages them.

Yes. You may want to reread the last paragraph.

It works directly in the disk image to avoid a nasty filesystem issue on emulated 32bit Linux over a 64bit mounted filesystem.

This feels like the most surreal Debian package I've ever created, and this saga looks like one of the hairiest yaks I've ever shaved.

Integrating this monster codebase, full of bundled code and hacks, into a streamlined production and deployment system has been for me a full stack nightmare, and I have a renewed and growing respect for the people in the Qt/KDE team in Debian, who manage to stay on top of this mess, so that it all just works when we need it.

Lite extra ball, from https://www.flickr.com/photos/st3f4n/143623902
Lite extra ball, from https://www.flickr.com/photos/st3f4n/143623902

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

The previous rounds of attempts ended in one issue too many to investigate in the allocated hourly budget.

Andreas Gruber wrote:

Long story short, a fast solution for the issue with EGLSetBlobFuncANDROID is to remove libraspberrypi-dev from your sysroot and do a full rebuild. There will be some changes to the configure results, so please review them - if they are relevant for you - before proceeding with your work.

That got me unstuck! dpkg --purge libraspberrypi-dev in the sysroot, and we're back in the game.

While Qt5's build has proven extremely fragile, I was surprised that some customization from Raspberry Pi hadn't yet broken something. In the end, they didn't disappoint.

More i386 issues

The run now stops with a new 32bit issue related to v8 snapshots:

qt-everywhere-src-5.15.0/qtwebengine/src/core/release$ /usr/bin/g++ -pie -Wl,--fatal-warnings -Wl,--build-id=sha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -m32 -pie -Wl,--disable-new-dtags -Wl,-O2 -Wl,--gc-sections -o "v8_snapshot/mksnapshot" -Wl,--start-group @"v8_snapshot/mksnapshot.rsp"  -Wl,--end-group  -ldl -lpthread -lrt -lz
/usr/bin/ld: skipping incompatible //usr/lib/x86_64-linux-gnu/libz.so when searching for -lz
/usr/bin/ld: skipping incompatible //usr/lib/x86_64-linux-gnu/libz.a when searching for -lz
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status

Attempted solution: apt install zlib1g-dev:i386.

Alternative solution (untried): configure Qt5 with -no-webengine-v8-snapshot.

It builds!

Installation paths

Now it tries to install files into debian/tmp/home/build/sysroot/opt/qt5custom-armhf/.

I realise that I now need to package the sysroot itself, both as a build-dependency of the Qt5 cross-compiler, and as a runtime dependency of the built cross-builder.

Conclusion

The current work in progress, patches, and all, is at https://github.com/Truelite/qt5custom/tree/master/debian-cross-qtwebengine

It blows my mind how ridiculously broken is the Qt5 cross-compiler build, for a use case that, looking at how many people are trying, seems to be one of the main ones for the cross-builder.

Question: what does this command do?

# Don't do this
nc localhost 12345 | sudo tar xf -

Answer: it sends the password typed into sudo to the other endpoint of netcat.

I can reproduce this with both nc.traditional and nc.openbsd.

One might be tempted to just put sudo in front of everything, but it'll mean that only nc will run as root:

# This is probably not what you want
sudo nc localhost 12345 | tar xf -

The fix that I will never remember, thanks to twb on IRC, is to close nc's stdin:

<&- nc localhost 12345 | sudo tar xf -

Or flip the table and just use sudo -s:

$ sudo -s
# nc localhost 12345 | tar xf -

Updates

Harald Koenig suggested two alternative spellings that might be easier to remember:

nc localhost 12345 < /dev/null | sudo tar xf -
< /dev/null nc localhost 12345 | sudo tar xf -

And thinking along those lines, there could also be the disappointed face variant:

:| nc localhost 12345 | sudo tar xf -

Matthias Urlichs suggested the approach of precaching sudo's credentials, making the rest of the command lines more straightforward (and TIL: sudo id):

sudo id
nc localhost 12345 | sudo tar xf -

Or even better:

sudo id && nc localhost 12345 | sudo tar xf -

Shortcomings of nc | tar

Tomas Janousek commented:

There's one more problem with a plain tar | nc | tar without redirection or extra parameteres: it doesn't know when to stop. So the best way to use it, I believe, is:

tar c | nc -N

nc -d | tar x

The -N option terminates the sending end of the connection, and the -d option tells the receiving netcat to never read any input. These two parameters, I hope, should also fix your sudo/password problem.

Hope it helps!

I noticed an odd effect, that reminds me of screen ghosting on old CRT monitors, when my laptop screen is locked:

Those faint white vertical lines one can see, are actually window borders, and the lock screen is leaking contents of my unlocked desktop! Here is the same screen, unlocked:

However, moving the windows around does not reflect on the ghost image on top of the lock screen: reshuffling windows then locking, produces always the same ghost image. The white border reflects where the window has been at some time in the past.

The lock screen does not seem to be responsible, either: dragging a solid colored window on the laptop screen has the same effect:

But taking a screenshot of it does not show the time traveling ghost windows:

This is happening on two different laptops, an HP EliteBook x360 G1, and a Lenovo ThinkPad X240, one that I've been using since 3 years, one that I've been using since a week, and whose only thing in common is a 1920x1080 IPS screen and an Intel GPU.

I have no idea where to start debugging this. Please reach out to me at enrico@debian.org if any of this makes sense to you.

Update: Jim Paris pointed me to https://en.wikipedia.org/wiki/Image_persistence which looks pretty much like what is happening here.

Jim Paris also pointed out that a black background doesn't show the ghosting.

I changed the lock screen background by editing /etc/lightdm/lightdm-gtk-greeter.conf and adding background=#000000 to the [greeter] section, to limit information leakage through ghosting in the lock screen.

Whack-A-Mole machines from <https://commons.wikimedia.org/wiki/File:Whac-A-Mole_Cedar_Point.jpg>
Whack-A-Mole machines from <https://commons.wikimedia.org/wiki/File:Whac-A-Mole_Cedar_Point.jpg>

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

Now that I have a sysroot, I try to use it to build Qt5 with QtWebEngine.

Nothing seems to work straightforwardly with Qt5's build system, and hit an endless series of significant blockers to try and work around.

(continue reading)

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi.

As an attempt to get webview to compile, I'm reattempting to build a Qt5 cross-compiling environment using a raspbian sysroot, instead of having dependencies for both arm and amd64 installed in the build system.

Using dependencies installed in a straightforward way in the build system has failed because of issues like #963136, where some of the build dependencies are needed for both architectures, but the corresponding Debian -dev packages are not yet coinstallable.

This is something that causes many people much pain.

(continue reading)