How to enable BFQ IO Scheduler on Ubuntu 19.04 – with Ansible playbook to set it up

I was trying to enable the BFQ scheduler on my laptop running Ubuntu 19.04, because as I moved around and created some large files my computer became basically unusable. After some investigation I found the page on the BFQ scheduler and how it is meant to address these sorts of issues.

There is some instructions on StackExchange on how to do this, so what I’ve done is turn it into an Ansible playbook to make it easier to apply. Here’s the original link: https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bfq-scheduler/376136

Note, this playbook doesn’t edit the /etc/default/grub file, because Ubuntu 19.04 already uses blk-mq by default. Here is the playbook:

To use the playbook, just save it somewhere and run with ansible-playbook, then reboot. For example:

~/l/a/bfq> ls
enable_bfq_playbook.yml
~/l/a/bfq> ansible-playbook enable_bfq_playbook.yml --ask-become-pass 
SUDO password: 
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] *************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [localhost]

TASK [Enable kernel module loading] ******************************************************************************************************************
ok: [localhost]

TASK [Enable bfq by default] *************************************************************************************************************************
ok: [localhost]

PLAY RECAP *******************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   

~/l/a/bfq> 

Running Nextcloud on the Raspberry Pi 3 – Nginx Reverse Proxy, fixes for upload timeouts and more

Raspberry Pi Nextcloud Server

Under my TV I have a Raspberry Pi 3 running Kodi, which works great for watching/listening to all my media. However, with it idling and always-on, I thought it would be great to be able to use it as a Nextcloud server as well. Here’s some details on my setup, as well as a some fixes for errors I encountered. Diving right in, here’s some details of my Pi’s OS install:

root@tvpi ###~> uname -a
Linux tvpi 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux
root@tvpi ###~> lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch
root@tvpi ###/etc> docker --version
Docker version 19.03.1, build 74b1e89

The setup I’m going for is:
HTTPS -> nginx proxy (in raspbian) -> nextcloud instance (in docker)

Setting up Nginx as a reverse proxy

The first part of the setup is to get nginx operating as a reverse proxy. The nginx server will do several things:

  • Redirect http traffic to https
  • Terminate https TLS traffic, and then proxy the traffic via http to the nextcloud server running in docker
  • Split traffic up based on the hostname used – to allow me to run other sites from the same Pi

To get a secure TLS configuration I am using Mozilla’s fantastic SSL Configuration Generator https://ssl-config.mozilla.org/. You then need some reverse proxy configuration, including setting X-Forwarded headers to allow the Nextcloud instance to block IP’s that are trying to brute force and also detect the correct hostname:

    location / {
        proxy_pass http://localhost:8088;
    }
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto 'https';
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

With my original setup I was seeing these errors:
"jtuckey/files/Documents/testfile.bin.upload.part" is locked
which would show as a 423 error in the Firefox dev-tools. According to Wikipedia this error is 423 Locked (WebDAV; RFC 4918). Here is screenshot of the problem:

These are the fixes I used for these issues with uploads, both with the maximum size of file uploads, and with file uploads timing out when I uploaded a large file. Nextcloud uploads large files in 10M chunks, and then re-combines them at the end into a single file. However, because of the Pi’s slow USB2 disk connection the re-combining can cause a timeout in the Nginx reverse proxy, which then re-tries the MOVE request, to which Nextcloud responds that the file is locked (it is already re-combining the file).

Also, when using the “file drop (upload only)” the uploads are not chunked, so to use that feature with large files you will need to set nginx’s max upload size to a very large value:

    # Allow uploads for NC - within the file app they upload in 10M chunks, however using file-drop
    # you need a value that is large enough for your largest file size
    client_max_body_size 5000M;

    # Allow long running connections, so NC can re-combine large files on a slow pi
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;

Here’s the final nginx configuration I have:

Running docker from an external USB SSD

To get extra storage and performance, I use a USB attached SSD connected to my pi, which you can see in the picture. This external disk is mounted at /mnt/tosh. To have docker put it’s storage on the external disk what I did is create a symbolic link from /var/lib/docker to the external disk:

root@tvpi ###/v/lib> pwd
/var/lib
root@tvpi ###/v/lib> ls -lah docker
lrwxrwxrwx 1 root root 18 Jun  9 09:11 docker -> /mnt/tosh/dockerd/

Make sure you haven’t got any data in docker that you want to keep, like existing volumes. Once you are happy, rename/remove the old /var/lib/docker directory and link it to the path you want to use. To create the link you can run:

root@tvpi ###/v/lib> ln -s /mnt/tosh/dockerd/ docker

Running the Nextcloud Docker image via docker-compose

The next part of the setup is to run the nextcloud docker image. The best way I found to do this was to use docker-compose. However, docker-compose doesn’t install on raspbian because it is arm32, so instead I installed it using pip, with the help of pyenv to install the latest version of python and pip. Pyenv is available here: https://github.com/pyenv/pyenv. To get the install requirements setup I had to run the line from the https://github.com/pyenv/pyenv/wiki page:

sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Once docker-compose is installed, I was able to use my docker-compose.yml file:

Note Line 15 – mapping in the extra config file for extending the apache2 timeout to help with file re-combining with large uploads. This is the contents of the apache_timeout.conf file:

# Timeout: The number of seconds before receives and sends time out.
Timeout 3600

Also take note of Lines 26 to 30 – where I set the IP Address Management. This allows setting the correct trusted_proxies value in the nextcloud config.php file.

Finally, also see in line 16 where I map in some extra external storage. I use Syncthing to do device synchronisation, because I find it’s syncing to be more reliable and efficient.

Running the Nextcloud cron job inside the docker container with a systemd timer

One situation the Nextcloud docker image has is that it doesn’t run cron within the container, so the nextcloud cron job isn’t regularly executed. To fix this, you need to run the job on a schedule on the host, to execute the cron.php file within the container. To do this, you can use cron, but I chose to use a systemd timer instead. The benefit this has is better logging and status checking. To create the config I used the excellent arch documentation here: https://wiki.archlinux.org/index.php/Systemd/Timers. Here are my nextcloud_cron.service and nextcloud_cron.timer files:

Setting the trusted_domains and trusted_proxies values in config.php

The final part of the configuration is to add a couple of extra values to the nextcloud config.php file. This should be added once you have started up the nextcloud container and run through the setup process. Once that’s complete, navigate to the nextcloud_config volume (will usually be at /var/lib/docker/volumes/nextcloud_config/_data) and add the extra values. Here’s what I added:

  'trusted_domains' =>  
  array (
    0 => 'localhost:8088',
    1 => 'nextcloud.jaytuckey.duckdns.org',
    2 => 'nc.jaytuckey.name',
  ),  
  'trusted_proxies' =>  
  array (
    0 => '172.16.0.1',
  ),  

Note the trusted_proxies value matches the subnet range we put in our docker-compose.yml file.

Bonus Points – An Ansible Playbook to configure most of these steps

To manage some of my configuration, I have created an ansible playbook to help set things up. Here’s my playbook:

How to get VMWare Remote Console to install on Kubuntu 19.04 – Probably works on other Ubuntu Versions also

When trying to set up my new Kubuntu install I went to install the VMWare Remote Console, and to my annoyance the installer would fail without any feedback on what was going wrong. Here’s what I needed to do to fix it. To start with , some details of my machines and setup:

Date: 2019-05-24
Machine details from inxi:
/m/u/h/j/S/notes> inxi
 CPU: Dual Core Intel Core i5-6200U (-MT MCP-) speed/min/max: 1468/400/2800 MHz Kernel: 5.0.0-15-generic x86_64 Up: 1h 49m 
 Mem: 4538.0/7719.6 MiB (58.8%) Storage: 238.47 GiB (54.3% used) Procs: 271 Shell: fish 3.0.2 inxi: 3.0.33

VMWare Remote Console Version: 10.0.4
VMware-Remote-Console-10.0.4-11818843.x86_64.bundle

To start with, running the installer from the command like I was getting this error:

~/d> sudo ./VMware-Remote-Console-10.0.4-11818843.x86_64.bundle --console
Extracting VMware Installer...done.
User interface initialization failed.  Exiting.  Check the log for details.

It wasn’t even clear what log file it meant, but after doing some searching on duckduckgo I discoversed the log file for the console installer is /var/log/vmware-installer. Therefore, to get an idea of the issues, I would tail that log like this:

root@jtuckey-x1-ubu:/var/log# tail -f /var/log/vmware-installer

Tailing that log showed me that there were some issues with libraries being used. To get the installer to run through I needed to install the packages:

  • libncursesw5 – for the console interface
  • desktop-file-utils – error caused when trying to run update-desktop-database command from this package

After this I was able to get the installed to run through completely. Hope this helps someone.

How to Play Age of Empires 3 on Ubuntu Linux using Steam Play – Solving the “Invalid CD Key – Error loading the PID Generator DLL”

This guide is based on one I found here: https://verybadfrags.com/2019/04/14/play-age-of-empires-iii-on-linux/

It was done with the following system:

Date: 2019-04-20
OS: Ubuntu 19.04
Steam Play Version: 4.2-3
Graphics: Nvidia GTX 860m with proprietary driver

This is to fix the error that you get when trying to run Age of Empires III: “Invalid CD Key! – Error loading the PID Generator DLL. The DLL could not be found! Please make sure the file is available in the installation directory and try again.”

  1. Enable steam play for all titles. Go to “Settings” -> “Steam Play” -> check “Enable Steam Play for all other titles”
  2. Launch Steam and install ‘Age of Empires III: Complete Collection’
  3. Run AoE3 for the first time, and let perform the first time setup. One you get to the “Product Key” box, click “Cancel”
  4. You now need to install “winetricks.” See https://github.com/Winetricks/winetricks. The easiest way to install on Ubuntu is to run sudo apt install winetricks
  5. You also need to install “protontricks”, which is a wrapper for winetricks that runs it against Steam Play installations. The version on the original article is out of date, so the newer version is here: https://github.com/Matoking/protontricks. To install this, run:
    1. sudo apt install python3-pip python3-setuptools python3-venv
    2. python3 -m pip install --user pipx
    3. ~/.local/bin/pipx ensurepath
    4. pipx install protontricks
  6. Now install the extra dependencies with protontricks: protontricks 105450 mfc42 winxp l3codecx corefonts
  7. Now you can relaunch the game and enter the CD Key.


How to Clean up VMware Horizon View pools without vCenter online and remove missing desktop pool from Global Entitlement using ADSI Edit

Recently I had to do a cleanup of a VMware Horizon 7 connection server, which involved removing all the existing desktop pools and recreating them. The trouble was, the old vCenter server had been removed, so when I tried to delete the pools using the Horizon Administrator Console, I got the error:

Server Error
Unable to connect to the vCenter Server

To fix this, I did the following:

Get the list of VM’s you want to remove

Using PowerCLI I was able to get a list of machines in the pool I wanted to remove. Install PowerCLI from the documentation here: https://docs.vmware.com/en/VMware-Horizon-7/7.6/horizon-integration/GUID-7C7C5239-6990-47E0-B9FB-29EC0EB0F5AC.html

Make sure to also install the VMware.Hv.Helper module from here https://github.com/vmware/PowerCLI-Example-Scripts by copying in to the C:\Program Files\WindowsPowerShell\Modules folder.

Then, after connecting to the Horizon View server, get a list of VM’s:

# Get all HV Machines
$ms = Get-HVMachine

# Show the HV Machine Names
$ms[0..2] | select -ExpandProperty Base | select Name

# Select just the machines you want
$to_remove = $ms | ?{$_.Base.Name -match '17a6-clst-p...'}

# To view a list of the machine names:
$to_remove | Select -ExpandProperty Base | Select name

# And export the list to a csv
$to_remove | Select -ExpandProperty Base | Select name | Export-Csv -NoTypeInformation ~\Downloads\to_remove.csv

Use SviConfig to delete machine records from the Horizon View Composer

This is from the documentation here: https://docs.vmware.com/en/VMware-Horizon-7/7.6/horizon-virtual-desktops/GUID-F0D595CB-4E7B-4DAE-B80B-DCDCE85E8DF2.html

Once you have copied the CSV to the composer server, you have to use the list to delete all the composer records for the machines.

# Run in an admin Powershell from path C:\Program Files (x86)\VMware\VMware View Composer
# Import the CSV
$to_delete = Import-Csv ~\Downloads\to_remove.csv

# Delete each item in the list
$to_delete | %{.\SviConfig.exe -operation=removesviclone -vmname="$($_.Name)" -AdminUser=<put admin account here> -AdminPassword="<put admin password here>"}

Remove the pool from any Global Entitlements

Important — This has to be done before removing the pool using ADSI Edit, in the next step. However, if you mess this up, as I did, I have a workaround as the last step.

To do this, just go into the global entitlements and remove the pool you are cleaning up from any of the global entitlements. If you don’t do this, you will see the global entitlement saying there are 2 pools in it, for example, but when you open the global entitlement to delete the pool you can’t see the local pool to remove.

Delete the pool using ADSI edit

To delete the actual pool and machine entries, follow the guide here: https://kb.vmware.com/s/article/2015112

The simple version is, you open ADSI edit to connect to server localhost and Distinguished Name/Naming Context dc=vdi,dc=vmware,dc=int.

You then create a query with the root of the search being OU=Servers,DC=vdi,DC=vmware,DC=int and the query string being (&(objectClass=pae-VM)(pae-displayname=17a6-clst-p*))

You can then check through the item the Applications and Server Groups OU’s to find the pool and delete it. However, make sure you have removed the pool from any global entitlements first.

Workaround – Delete Global Entitlement Local Pool member using ADSI edit

To do this, open ADSI edit on the connection server, and choose “Connect to”. Use localhost:22389 as the server, and DC=vdiglobal,DC=vmware,DC=int as the Distinguished Name/Naming Context.

Create a new query, as you did with deleting the pool using ADSI edit. This query should be in the new connection, and have the settings:

Name: Find global entitlement \
Root of search: OU=Entitlements,DC=vdiglobal,DC=vmware,DC=int \
Query String: (&(objectClass=pae-GlobalAssignment)(pae-LocalEntitlement=*17a6-clst-p*))

Note that the name of the local pool you haven’t removed from the global entitlement is in the query between the * characters.

This should give you one item, of type pae-GlobalAssignment. Open it up, and make sure the pae-LocalEntitlement attribute matches what you want to delete. If so, delete it.

You now shouldn’t have the incorrect number of local pools in your global entitlement.

Using xonsh shell with pyenv on Ubuntu 18.04 – and a few errors I had getting source-bash to work 😉

When I heard about the xonsh project – https://xon.sh I thought it sounded great, as I really enjoy the python language and am much more comfortable using python to write scripts than I am using bash. I also find the bash syntax very confusing, and am never sure of the right way to write a script or use a variable.

So, I decided to install it and give it a try! When using python on Linux I always use the pyenv tool to install and manage my python versions, as well as my virtualenvs. This tool allows me to easily install the latest version of python, as well as create a virtualenv for a specific project and keep all the packages that project uses separate from each other and my system python. It’s a great tool, and I can highly recommend using it.

pyenv installs all python versions into your user profile, and allows you to select from them on the command line using

To get all the pyenv commands, along with the pyenv command completion (which is very good), you add an init command to the end of your .bashrc file. This is how it looks in mine:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Excellent! When running in bash this sets up pyenv, then sets my shell to use python 3.7.0. So let’s install xonsh:

jtuckey@jay-x1-carbon:~$ pip install --user xonsh
Collecting xonsh
  Using cached https://files.pythonhosted.org/packages/58/16/fce8ecc880a44dfcb06b22f638df0b6982ad588eb0f2080fbd5218a42340/xonsh-0.8.0.tar.gz
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.8.0

Looking good so far! I can now use the xonsh command to hop into a xonsh shell, and then use all the goodness of a python-based shell:

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ $ ${'XONSH_VERSION'}
'0.8.0'
jtuckey@jay-x1-carbon ~ $ history info
backend: json
sessionid: 30e7e6d4-4945-4f99-9b65-d38a2ab61da2
filename: /home/jtuckey/.local/share/xonsh/xonsh-30e7e6d4-4945-4f99-9b65-d38a2ab61da2.json
length: 1
buffersize: 100
bufferlength: 1
gc options: (8128, 'commands')

Now that I have it installed and am getting all the good features, I naturally wanted it as my default shell. My first attempt at this was to simply put it the end of my .bashrc file:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

xonsh

This works! However, to quit the shell, I need to exit twice:

jtuckey@jay-x1-carbon ~ <xonsh>$ exit
jtuckey@jay-x1-carbon:~$ exit

Ok, maybe let’s put it in an auto-exiting block. Back to the .bashrc:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
if xonsh; then
  exit
fi

This works even better. Now it will automatically exit when I’m done using xonsh, but if it can’t find xonsh it will stay in bash. However, with this setup, I try to do some work, which involves activating one of my pyenv virtualenvs. This gives me an error:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Ok, so clearly the pyenv setup chunk from bash isn’t carrying across nicely from bash to xonsh. The path looks alright:

jtuckey@jay-x1-carbon ~ <xonsh>$ $PATH
\EnvPath(
['/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.local/bin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']
)

Let’s have a look at what those two pyenv init commands. What they are doing is generating a bit of code using pyenv init - pyenv virtualenv-init - and then evaluating it in the local scope. We can easily see the code that’s being generated:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv init -
export PATH="/home/jtuckey/.pyenv/shims:${PATH}"
export PYENV_SHELL=python3.7
command pyenv rehash 2>/dev/null
pyenv() {
  local command
  command="${1:-}"
  if [ "$#" -gt 0 ]; then
    shift
  fi

  case "$command" in
  activate|deactivate|rehash|shell)
    eval "$(pyenv "sh-$command" "$@")";;
  *)
    command pyenv "$command" "$@";;
  esac
}
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv virtualenv-init -
export PATH="/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
  local ret=$?
  if [ -n "$VIRTUAL_ENV" ]; then
    eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
  else
    eval "$(pyenv sh-activate --quiet || true)" || true
  fi
  return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
  PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi

Ok, so if we can load these as bash, we can get all the commands correctly into xonsh. Fortunately there is a command in xonsh just for this:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash --help
usage: source-foreign [-h] [-i INTERACTIVE] [-l LOGIN] [--envcmd ENVCMD]
                      [--aliascmd ALIASCMD] [--extra-args EXTRA_ARGS]
                      [-s SAFE] [-p PREVCMD] [--postcmd POSTCMD]
                      [--funcscmd FUNCSCMD] [--sourcer SOURCER]
                      [--use-tmpfile USE_TMPFILE]
                      [--seterrprevcmd SETERRPREVCMD]
                      [--seterrpostcmd SETERRPOSTCMD] [--overwrite-aliases]
                      [--suppress-skip-message] [--show] [-d]
                      shell files_or_code [files_or_code ...]

Sources a file written in a foreign shell language.

positional arguments:
  shell                 Name or path to the foreign shell
  files_or_code         file paths to source or code in the target language.
...

jtuckey@jay-x1-carbon ~ <xonsh>$ aliases
xonsh.aliases.Aliases(
{...
 'source-bash': ['source-foreign', 'bash', '--sourcer=source']
...}

So we should be able to just source the output of the pyenv commands. Let’s try it:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
# Shell just hangs at this point....

Sooo…. that’s not working. My shell just locks up. I can’t even Ctrl-C to kill it.

What’s going on here? After a bit of investigation, what seems to be happening is this: when you use source-bash the bash shell launched to run the source is running the .bashrc file, which puts it back into xonsh, where it gets stuck. This makes sense, although it’s hard to work out. To test this, lets get rid of the xonsh lines from .bashrc and see if it works correctly then.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
#if xonsh; then
#  exit
#fi

And try running it manually

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ 

Looking better. Now I’m back where I started, though, so how can I run xonsh as my shell. From the xonsh tutorial there is this line:

Alternatively, you can setup your terminal emulator (xterm, gnome-terminal, etc) to run xonsh automatically when it starts up. This is recommended.

However, in my gnome-terminal if I try setting xonsh as the command it can’t find xonsh. This is because the python 3.7 environment where xonsh lives is initialised during bash startup, not gui startup, so my gnome-terminal can’t find the xonsh command.

Ok, so how do I put something into my whole user environment? That’s what the .profile file in your user profile is for. Let’s put the pyenv init stuff into the end of my .profile file:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Now with a logoff and back on, let’s see what happens. Now when I set gnome-terminal to run xonsh, it works. However:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.


jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv virtualenv-init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Hmmm, still not quite there, but closer.

When I heard about the xonsh project – https://xon.sh I thought it sounded great, as I really enjoy the python language and am much more comfortable using python to write scripts than I am using bash. I also find the bash syntax very confusing, and am never sure of the right way to write a script or use a variable.

So, I decided to install it and give it a try! When using python on Linux I always use the pyenv tool to install and manage my python versions, as well as my virtualenvs. This tool allows me to easily install the latest version of python, as well as create a virtualenv for a specific project and keep all the packages that project uses separate from each other and my system python. It’s a great tool, and I can highly recommend using it.

pyenv installs all python versions into your user profile, and allows you to select from them on the command line using

pyenv shell 3.7.0

To get all the pyenv commands, along with the pyenv command completion (which is very good), you add an init command to the end of your .bashrc file. This is how it looks in mine:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Excellent! When running in bash this sets up pyenv, then sets my shell to use python 3.7.0. So let’s install xonsh:

jtuckey@jay-x1-carbon:~$ pip install --user xonsh
Collecting xonsh
  Using cached https://files.pythonhosted.org/packages/58/16/fce8ecc880a44dfcb06b22f638df0b6982ad588eb0f2080fbd5218a42340/xonsh-0.8.0.tar.gz
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.8.0

Looking good so far! I can now use the xonsh command to hop into a xonsh shell, and then use all the goodness of a python-based shell:

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ $ ${'XONSH_VERSION'}
'0.8.0'
jtuckey@jay-x1-carbon ~ $ history info
backend: json
sessionid: 30e7e6d4-4945-4f99-9b65-d38a2ab61da2
filename: /home/jtuckey/.local/share/xonsh/xonsh-30e7e6d4-4945-4f99-9b65-d38a2ab61da2.json
length: 1
buffersize: 100
bufferlength: 1
gc options: (8128, 'commands')

Now that I have it installed and am getting all the good features, I naturally wanted it as my default shell. My first attempt at this was to simply put it the end of my .bashrc file:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

xonsh

This works! However, to quit the shell, I need to exit twice:

jtuckey@jay-x1-carbon ~ <xonsh>$ exit
jtuckey@jay-x1-carbon:~$ exit

Ok, maybe let’s put it in an auto-exiting block. Back to the .bashrc:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
if xonsh; then
  exit
fi

This works even better. Now it will automatically exit when I’m done using xonsh, but if it can’t find xonsh it will stay in bash. However, with this setup, I try to do some work, which involves activating one of my pyenv virtualenvs. This gives me an error:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Ok, so clearly the pyenv setup chunk from bash isn’t carrying across nicely from bash to xonsh. The path looks alright:

jtuckey@jay-x1-carbon ~ <xonsh>$ $PATH
\EnvPath(
['/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.local/bin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']
)

Let’s have a look at what those two pyenv init commands. What they are doing is generating a bit of code using pyenv init - pyenv virtualenv-init - and then evaluating it in the local scope. We can easily see the code that’s being generated:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv init -
export PATH="/home/jtuckey/.pyenv/shims:${PATH}"
export PYENV_SHELL=python3.7
command pyenv rehash 2>/dev/null
pyenv() {
  local command
  command="${1:-}"
  if [ "$#" -gt 0 ]; then
    shift
  fi

  case "$command" in
  activate|deactivate|rehash|shell)
    eval "$(pyenv "sh-$command" "$@")";;
  *)
    command pyenv "$command" "$@";;
  esac
}
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv virtualenv-init -
export PATH="/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
  local ret=$?
  if [ -n "$VIRTUAL_ENV" ]; then
    eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
  else
    eval "$(pyenv sh-activate --quiet || true)" || true
  fi
  return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
  PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi

Ok, so if we can load these as bash, we can get all the commands correctly into xonsh. Fortunately there is a command in xonsh just for this:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash --help
usage: source-foreign [-h] [-i INTERACTIVE] [-l LOGIN] [--envcmd ENVCMD]
                      [--aliascmd ALIASCMD] [--extra-args EXTRA_ARGS]
                      [-s SAFE] [-p PREVCMD] [--postcmd POSTCMD]
                      [--funcscmd FUNCSCMD] [--sourcer SOURCER]
                      [--use-tmpfile USE_TMPFILE]
                      [--seterrprevcmd SETERRPREVCMD]
                      [--seterrpostcmd SETERRPOSTCMD] [--overwrite-aliases]
                      [--suppress-skip-message] [--show] [-d]
                      shell files_or_code [files_or_code ...]

Sources a file written in a foreign shell language.

positional arguments:
  shell                 Name or path to the foreign shell
  files_or_code         file paths to source or code in the target language.
...

jtuckey@jay-x1-carbon ~ <xonsh>$ aliases
xonsh.aliases.Aliases(
{...
 'source-bash': ['source-foreign', 'bash', '--sourcer=source']
...}

So we should be able to just source the output of the pyenv commands. Let’s try it:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
# Shell just hangs at this point....

Sooo…. that’s not working. My shell just locks up. I can’t even Ctrl-C to kill it.

What’s going on here? After a bit of investigation, what seems to be happening is this: when you use source-bash the bash shell launched to run the source is running the .bashrc file, which puts it back into xonsh, where it gets stuck. This makes sense, although it’s hard to work out. To test this, lets get rid of the xonsh lines from .bashrc and see if it works correctly then.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
#if xonsh; then
#  exit
#fi

And try running it manually

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ 

Looking better. Now I’m back where I started, though, so how can I run xonsh as my shell. From the xonsh tutorial there is this line:

Alternatively, you can setup your terminal emulator (xterm, gnome-terminal, etc) to run xonsh automatically when it starts up. This is recommended.

However, in my gnome-terminal if I try setting xonsh as the command it can’t find xonsh. This is because the python 3.7 environment where xonsh lives is initialised during bash startup, not gui startup, so my gnome-terminal can’t find the xonsh command.

Ok, so how do I put something into my whole user environment? That’s what the .profile file in your user profile is for. Let’s put the pyenv init stuff into the end of my .profile file:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Now with a logoff and back on, let’s see what happens. Now when I set gnome-terminal to run xonsh, it works. However:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.


jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv virtualenv-init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Hmmm, still not quite there, but closer. After some further research, I found this on a page about virtual environments and xonsh.

The usual tools for creating Python virtual environments—venv, virtualenv, pew—don’t play well with xonsh. …. Luckily, xonsh ships with its own virtual environments manager called Vox.

https://xon.sh/python_virtual_environments.html

Sounds like vox is the way to go. To load vox you just need to load it. However, by default it doesn’t look for virtualenvs in the pyenv path. This is easily fixable by setting the $VIRTUALENV_HOME variable:

jay@jay-alienware-ubuntu:~:x$ xontrib load vox                                                                               
jay@jay-alienware-ubuntu:~:x$ vox list                                                                                       
No environments available. Create one with "vox new".

jay@jay-alienware-ubuntu:~:x$ $VIRTUALENV_HOME = $HOME +'/.pyenv/versions'                                                   
jay@jay-alienware-ubuntu:~:x$ vox list                                                                                       
Available environments:
3.6.2/envs/test-install
3.6.2/envs/tests
3.6.4/envs/mdb
3.7.0/envs/xonshdev
jay@jay-alienware-ubuntu:~:x$ vox activate tests                                                                             
Activated "tests".

(tests) jay@jay-alienware-ubuntu:~:x$ vox deactivate                                                                         
Deactivated "tests".

jay@jay-alienware-ubuntu:~:x$  

Be aware that creating a virtualenv through vox will just use your currently set python version from pyenv. However, you can still use pyenv to create a new virtualenv of any version, and then use vox to activate it:

jay@jay-alienware-ubuntu:~:x$ pyenv virtualenv 3.6.4 newenv                                                                  
Requirement already satisfied: setuptools in /home/jay/.pyenv/versions/3.6.4/envs/newenv/lib/python3.6/site-packages
Requirement already satisfied: pip in /home/jay/.pyenv/versions/3.6.4/envs/newenv/lib/python3.6/site-packages
jay@jay-alienware-ubuntu:~:x$ vox activate newenv                                                                            
Activated "newenv".

(newenv) jay@jay-alienware-ubuntu:~:x$  

So I now have a fully working pyenv+xonsh setup! The only thing that’s missing is proper tab-completion of the pyenv commands, which is something I may look into in the future, but for now I’m able to do my work easily without leaving the xonsh shell, which is great!

Fix for Windows Server 2016 Stuck at “Downloading 0%” from WSUS server – UpdateServiceUrlAlternate Registry Key

I ran into an issue recently with our Windows Server 2016 servers not installing their updates. Instead, all our 2016 servers would get stuck with the message “Updates are available. Downloading updates 0%” however they would never actually download the updates. These servers were configured to our WSUS server, which was also a Server 2016 machine.

The first thing I investigated was to look at the logs on the client server. I tried using the Get-WindowsUpdateLog cmdlet in Powershell, however, the log that was generated was useless because every line had no date and no format information (what the hell, Microsoft!). The logs look like this:

1601/01/01 09:30:00.0000000 872   1104                  Unknown( 39): GUID=50a5ef43-b2b6-3c52-145b-a386d8769665 (No Format Information found).
1601/01/01 09:30:00.0000000 872   1104                  Unknown( 38): GUID=50a5ef43-b2b6-3c52-145b-a386d8769665 (No Format Information found).
1601/01/01 09:30:00.0000000 872   1104                  Unknown( 34): GUID=7b9bf239-47b9-3688-3a9e-14f09f262608 (No Format Information found).
1601/01/01 09:30:00.0000000 872   1104                  Unknown( 39): GUID=50a5ef43-b2b6-3c52-145b-a386d8769665 (No Format Information found).
1601/01/01 09:30:00.0000000 872   1104                  Unknown( 38): GUID=50a5ef43-b2b6-3c52-145b-a386d8769665 (No Format Information found).
1601/01/01 09:30:00.0000000 872   1104                  Unknown( 34): GUID=7b9bf239-47b9-3688-3a9e-14f09f262608 (No Format Information found).
1601/01/01 09:30:00.0000000 872   1104                  Unknown( 39): GUID=50a5ef43-b2b6-3c52-145b-a386d8769665 (No Format Information found).

However, after investigating the registry keys on one of our servers, I saw a new registry key that I didn’t recognise, UpdateServiceUrlAlternate. See this dump from Powershell:

PS C:\Users\me\Desktop> ls HKLM:\SOFTWARE\Policies\Microsoft\Windows\


    Hive: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows


Name                           Property
----                           --------
Appx
BITS                           EnableBitsMaxBandwidth     : 0
                               MaxBandwidthValidFrom      : 9
                               MaxBandwidthValidTo        : 17
                               MaxTransferRateOnSchedule  : 1000
                               MaxTransferRateOffSchedule : 9999
CurrentVersion
DataCollection
EnhancedStorageDevices         TCGSecurityActivationDisabled : 0
IPSec
Network Connections            NC_PersonalFirewallConfig : 0
NetworkConnectivityStatusIndic (default) :
ator
NetworkProvider
safer
SettingSync                    EnableBackupForWin8Apps : 1
System
WcmSvc
WindowsUpdate                  UpdateServiceUrlAlternate                    : http://localhost:8005
                               WUServer                                     : http://wsusserver.domainname.au:8530
                               WUStatusServer                               : http://wsus.domainname.au:8530
                               ElevateNonAdmins                             : 0
                               DoNotConnectToWindowsUpdateInternetLocations : 1
                               TargetGroupEnabled                           : 1
                               TargetGroup                                  : Auto Patch2
WorkplaceJoin                  (default) :
WSDAPI

After investigating this for a while, I found that the UpdateServiceUrlAlternate is a new setting introduced in the 2017 January update for Windows 10 1607 (and Server 2016, it would seem). For more details, see the Microsoft Documentation for UpdateServiceUrlAlternate.

We discovered that we had not been setting this with our current Group Policy, because we were using ADMX templates that were released before this was added. After updating our ADMX templates and setting that value to nothing, the updates download and install correctly again.

VMware User Environment Manager (UEM) Profile for Autodesk AutoCAD 2017 in VDI

In our Horizon View VDI environment we use UEM to manage some application’s user data. I used the Application Profiler included with UEM to create this profile for AutoCAD. One issue that we we have with AutoCAD is that it will reconfigure when launched by a new user. We tried to capture a default profile but the registry entries captured include lots of references to the specific user profile path for that user, which caused issues on launch. I have attached the INI file and icons to this post.

To prevent AutoCAD from crashing on launch, make sure to capture <LocalAppData>\Autodesk, which isn’t included by default when using the Application Profiler.

Application: Autodesk AutoCAD 2017

OS: Windows 7

INI Details:

# Flex config file generated with VMware User Environment Manager Application Profiler version 9.1

[IncludeRegistryTrees]
HKCU\Software\Autodesk
HKCU\SOFTWARE\FLEXlm License Manager

[IncludeFolderTrees]
<AppData>\Autodesk
# Added to prevent crash on launch
<LocalAppData>\Autodesk

[Metadata]
ProcessIfExists=
DirectFlexEnabled
DirectFlexPath=%ProgramFiles%\Autodesk\AutoCAD 2017\acad.exe
ProcessCriteria=
ProfilerChild=c:\program files\autodesk\autocad 2017\acwebbrowser\acwebbrowser.exe
ProfilerChild=c:\program files\autodesk\autocad 2017\admigrator.exe
ProfilerChild=c:\windows\microsoft.net\framework64\v4.0.30319\csc.exe
ProfilerChild=c:\windows\microsoft.net\framework64\v4.0.30319\cvtres.exe
ProfilerChild=c:\program files\autodesk\autocad 2017\dadispatcherservice.exe


[Immidio Flex]
This file was created using VMware UEM Management Console version 9.1.0.175.
Use only with VMware UEM.

Zipped INI File and Icon: Autodesk_AutoCAD_2017

Please feel free to comment with any issues or suggestions you had using AutoCAD with UEM in your environment.

Experiment: Using the Colemak Keyboard Layout For Two Weeks

Recently I have been investigating using better, more ergonomic computer equipment. To this end, several months ago I purchased an ergonomic keyboard, the Microsoft Natural Ergonomic Keyboard 4000. During my investigation of keyboards, however, I came across a lot of discussion about alternative keyboard layouts, and their various pros and cons, and have decided to try switching to the Colemak Keyboard layout.

About Me

I don’t have any issues with RSI, or pain or stiffness in my hands. I would like to keep it that way, and so I would like to use good ergonomic equipment and practices to keep my hands that way. I work in IT, and do a large amount of typing throughout the day, so even a small improvement to the quality of my typing can pay off for me.

Choosing Colemak

During my investigation on different keyboard layouts, I of course found a lot of information on Dvorak, another popular alternative layout. Certainly there are some significant benefits to Dvorak:

  1. It has more OS support (comes built-in on Windows)
  2. It’s more widely used, and therefore more studied.

So why choose Colemak?

  1. Most importantly for me, it keeps some of the keyboard shortcuts the same, like Ctrl-C, Ctrl-X and Ctrl-V. On Dvorak the common pattern Ctrl-C -> Ctrl-V are both two handed shortcuts. 🙁
  2. It has slightly less finger travel than Dvorak. For more details, see https://colemak.com/Ergonomic
  3. It keeps almost all the symbol keys in the same place, which is nice.

Day 1

The first day was so hard, and it went soooo slowly. Very frustrating, and by the end of the day I was getting only 12 WPM. I found I was constantly having to read off the image of the keyboard layout to type simple things.

Days 2 – 7

Once I got through the first day, I slowly improved over the next few days. I found that I was not having to look to remember the keys on the home row, and only check for keys above or below. However, even with the home row, I would get stuck with a mental block about once per sentence, trying to un-jumble the keys in my mind. This significantly slowed my typing, but as the week went on, the pauses would get shorter and less common.

Days 8 – 14

Once I was past the first week, I found I was no longer running into issues getting stuck, however my accuracy and speed were still lower than I had on QWERTY. However, as the week progresses I found my speed and accuracy picking up. I also started using the CapsLock backspace key, which is great.

Conclusion

So how fast am I typing now? I just tested my typing speed, and I’m getting 35 WPM. That’s not bad, but it’s not great. I have previously scored up to 50 WPM on QWERTY, although I haven’t tested my speed for a while. However, I plan to continue using Colemak, as I find it very comfortable to type on. We’ll see how I’m going after a couple of months of usage.