How to enable BFQ IO Scheduler on Ubuntu 19.04 – with Ansible playbook to set it up

I was trying to enable the BFQ scheduler on my laptop running Ubuntu 19.04, because as I moved around and created some large files my computer became basically unusable. After some investigation I found the page on the BFQ scheduler and how it is meant to address these sorts of issues.

There is some instructions on StackExchange on how to do this, so what I’ve done is turn it into an Ansible playbook to make it easier to apply. Here’s the original link: https://unix.stackexchange.com/questions/375600/how-to-enable-and-use-the-bfq-scheduler/376136

Note, this playbook doesn’t edit the /etc/default/grub file, because Ubuntu 19.04 already uses blk-mq by default. Here is the playbook:

To use the playbook, just save it somewhere and run with ansible-playbook, then reboot. For example:

~/l/a/bfq> ls
enable_bfq_playbook.yml
~/l/a/bfq> ansible-playbook enable_bfq_playbook.yml --ask-become-pass 
SUDO password: 
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'


PLAY [localhost] *************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [localhost]

TASK [Enable kernel module loading] ******************************************************************************************************************
ok: [localhost]

TASK [Enable bfq by default] *************************************************************************************************************************
ok: [localhost]

PLAY RECAP *******************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   

~/l/a/bfq> 

Running Nextcloud on the Raspberry Pi 3 – Nginx Reverse Proxy, fixes for upload timeouts and more

Raspberry Pi Nextcloud Server

Under my TV I have a Raspberry Pi 3 running Kodi, which works great for watching/listening to all my media. However, with it idling and always-on, I thought it would be great to be able to use it as a Nextcloud server as well. Here’s some details on my setup, as well as a some fixes for errors I encountered. Diving right in, here’s some details of my Pi’s OS install:

root@tvpi ###~> uname -a
Linux tvpi 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux
root@tvpi ###~> lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch
root@tvpi ###/etc> docker --version
Docker version 19.03.1, build 74b1e89

The setup I’m going for is:
HTTPS -> nginx proxy (in raspbian) -> nextcloud instance (in docker)

Setting up Nginx as a reverse proxy

The first part of the setup is to get nginx operating as a reverse proxy. The nginx server will do several things:

  • Redirect http traffic to https
  • Terminate https TLS traffic, and then proxy the traffic via http to the nextcloud server running in docker
  • Split traffic up based on the hostname used – to allow me to run other sites from the same Pi

To get a secure TLS configuration I am using Mozilla’s fantastic SSL Configuration Generator https://ssl-config.mozilla.org/. You then need some reverse proxy configuration, including setting X-Forwarded headers to allow the Nextcloud instance to block IP’s that are trying to brute force and also detect the correct hostname:

    location / {
        proxy_pass http://localhost:8088;
    }
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto 'https';
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

With my original setup I was seeing these errors:
"jtuckey/files/Documents/testfile.bin.upload.part" is locked
which would show as a 423 error in the Firefox dev-tools. According to Wikipedia this error is 423 Locked (WebDAV; RFC 4918). Here is screenshot of the problem:

These are the fixes I used for these issues with uploads, both with the maximum size of file uploads, and with file uploads timing out when I uploaded a large file. Nextcloud uploads large files in 10M chunks, and then re-combines them at the end into a single file. However, because of the Pi’s slow USB2 disk connection the re-combining can cause a timeout in the Nginx reverse proxy, which then re-tries the MOVE request, to which Nextcloud responds that the file is locked (it is already re-combining the file).

Also, when using the “file drop (upload only)” the uploads are not chunked, so to use that feature with large files you will need to set nginx’s max upload size to a very large value:

    # Allow uploads for NC - within the file app they upload in 10M chunks, however using file-drop
    # you need a value that is large enough for your largest file size
    client_max_body_size 5000M;

    # Allow long running connections, so NC can re-combine large files on a slow pi
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;

Here’s the final nginx configuration I have:

Running docker from an external USB SSD

To get extra storage and performance, I use a USB attached SSD connected to my pi, which you can see in the picture. This external disk is mounted at /mnt/tosh. To have docker put it’s storage on the external disk what I did is create a symbolic link from /var/lib/docker to the external disk:

root@tvpi ###/v/lib> pwd
/var/lib
root@tvpi ###/v/lib> ls -lah docker
lrwxrwxrwx 1 root root 18 Jun  9 09:11 docker -> /mnt/tosh/dockerd/

Make sure you haven’t got any data in docker that you want to keep, like existing volumes. Once you are happy, rename/remove the old /var/lib/docker directory and link it to the path you want to use. To create the link you can run:

root@tvpi ###/v/lib> ln -s /mnt/tosh/dockerd/ docker

Running the Nextcloud Docker image via docker-compose

The next part of the setup is to run the nextcloud docker image. The best way I found to do this was to use docker-compose. However, docker-compose doesn’t install on raspbian because it is arm32, so instead I installed it using pip, with the help of pyenv to install the latest version of python and pip. Pyenv is available here: https://github.com/pyenv/pyenv. To get the install requirements setup I had to run the line from the https://github.com/pyenv/pyenv/wiki page:

sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Once docker-compose is installed, I was able to use my docker-compose.yml file:

Note Line 15 – mapping in the extra config file for extending the apache2 timeout to help with file re-combining with large uploads. This is the contents of the apache_timeout.conf file:

# Timeout: The number of seconds before receives and sends time out.
Timeout 3600

Also take note of Lines 26 to 30 – where I set the IP Address Management. This allows setting the correct trusted_proxies value in the nextcloud config.php file.

Finally, also see in line 16 where I map in some extra external storage. I use Syncthing to do device synchronisation, because I find it’s syncing to be more reliable and efficient.

Running the Nextcloud cron job inside the docker container with a systemd timer

One situation the Nextcloud docker image has is that it doesn’t run cron within the container, so the nextcloud cron job isn’t regularly executed. To fix this, you need to run the job on a schedule on the host, to execute the cron.php file within the container. To do this, you can use cron, but I chose to use a systemd timer instead. The benefit this has is better logging and status checking. To create the config I used the excellent arch documentation here: https://wiki.archlinux.org/index.php/Systemd/Timers. Here are my nextcloud_cron.service and nextcloud_cron.timer files:

Setting the trusted_domains and trusted_proxies values in config.php

The final part of the configuration is to add a couple of extra values to the nextcloud config.php file. This should be added once you have started up the nextcloud container and run through the setup process. Once that’s complete, navigate to the nextcloud_config volume (will usually be at /var/lib/docker/volumes/nextcloud_config/_data) and add the extra values. Here’s what I added:

  'trusted_domains' =>  
  array (
    0 => 'localhost:8088',
    1 => 'nextcloud.jaytuckey.duckdns.org',
    2 => 'nc.jaytuckey.name',
  ),  
  'trusted_proxies' =>  
  array (
    0 => '172.16.0.1',
  ),  

Note the trusted_proxies value matches the subnet range we put in our docker-compose.yml file.

Bonus Points – An Ansible Playbook to configure most of these steps

To manage some of my configuration, I have created an ansible playbook to help set things up. Here’s my playbook:

How to get VMWare Remote Console to install on Kubuntu 19.04 – Probably works on other Ubuntu Versions also

When trying to set up my new Kubuntu install I went to install the VMWare Remote Console, and to my annoyance the installer would fail without any feedback on what was going wrong. Here’s what I needed to do to fix it. To start with , some details of my machines and setup:

Date: 2019-05-24
Machine details from inxi:
/m/u/h/j/S/notes> inxi
 CPU: Dual Core Intel Core i5-6200U (-MT MCP-) speed/min/max: 1468/400/2800 MHz Kernel: 5.0.0-15-generic x86_64 Up: 1h 49m 
 Mem: 4538.0/7719.6 MiB (58.8%) Storage: 238.47 GiB (54.3% used) Procs: 271 Shell: fish 3.0.2 inxi: 3.0.33

VMWare Remote Console Version: 10.0.4
VMware-Remote-Console-10.0.4-11818843.x86_64.bundle

To start with, running the installer from the command like I was getting this error:

~/d> sudo ./VMware-Remote-Console-10.0.4-11818843.x86_64.bundle --console
Extracting VMware Installer...done.
User interface initialization failed.  Exiting.  Check the log for details.

It wasn’t even clear what log file it meant, but after doing some searching on duckduckgo I discoversed the log file for the console installer is /var/log/vmware-installer. Therefore, to get an idea of the issues, I would tail that log like this:

root@jtuckey-x1-ubu:/var/log# tail -f /var/log/vmware-installer

Tailing that log showed me that there were some issues with libraries being used. To get the installer to run through I needed to install the packages:

  • libncursesw5 – for the console interface
  • desktop-file-utils – error caused when trying to run update-desktop-database command from this package

After this I was able to get the installed to run through completely. Hope this helps someone.

Using xonsh shell with pyenv on Ubuntu 18.04 – and a few errors I had getting source-bash to work 😉

When I heard about the xonsh project – https://xon.sh I thought it sounded great, as I really enjoy the python language and am much more comfortable using python to write scripts than I am using bash. I also find the bash syntax very confusing, and am never sure of the right way to write a script or use a variable.

So, I decided to install it and give it a try! When using python on Linux I always use the pyenv tool to install and manage my python versions, as well as my virtualenvs. This tool allows me to easily install the latest version of python, as well as create a virtualenv for a specific project and keep all the packages that project uses separate from each other and my system python. It’s a great tool, and I can highly recommend using it.

pyenv installs all python versions into your user profile, and allows you to select from them on the command line using

To get all the pyenv commands, along with the pyenv command completion (which is very good), you add an init command to the end of your .bashrc file. This is how it looks in mine:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Excellent! When running in bash this sets up pyenv, then sets my shell to use python 3.7.0. So let’s install xonsh:

jtuckey@jay-x1-carbon:~$ pip install --user xonsh
Collecting xonsh
  Using cached https://files.pythonhosted.org/packages/58/16/fce8ecc880a44dfcb06b22f638df0b6982ad588eb0f2080fbd5218a42340/xonsh-0.8.0.tar.gz
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.8.0

Looking good so far! I can now use the xonsh command to hop into a xonsh shell, and then use all the goodness of a python-based shell:

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ $ ${'XONSH_VERSION'}
'0.8.0'
jtuckey@jay-x1-carbon ~ $ history info
backend: json
sessionid: 30e7e6d4-4945-4f99-9b65-d38a2ab61da2
filename: /home/jtuckey/.local/share/xonsh/xonsh-30e7e6d4-4945-4f99-9b65-d38a2ab61da2.json
length: 1
buffersize: 100
bufferlength: 1
gc options: (8128, 'commands')

Now that I have it installed and am getting all the good features, I naturally wanted it as my default shell. My first attempt at this was to simply put it the end of my .bashrc file:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

xonsh

This works! However, to quit the shell, I need to exit twice:

jtuckey@jay-x1-carbon ~ <xonsh>$ exit
jtuckey@jay-x1-carbon:~$ exit

Ok, maybe let’s put it in an auto-exiting block. Back to the .bashrc:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
if xonsh; then
  exit
fi

This works even better. Now it will automatically exit when I’m done using xonsh, but if it can’t find xonsh it will stay in bash. However, with this setup, I try to do some work, which involves activating one of my pyenv virtualenvs. This gives me an error:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Ok, so clearly the pyenv setup chunk from bash isn’t carrying across nicely from bash to xonsh. The path looks alright:

jtuckey@jay-x1-carbon ~ <xonsh>$ $PATH
\EnvPath(
['/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.local/bin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']
)

Let’s have a look at what those two pyenv init commands. What they are doing is generating a bit of code using pyenv init - pyenv virtualenv-init - and then evaluating it in the local scope. We can easily see the code that’s being generated:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv init -
export PATH="/home/jtuckey/.pyenv/shims:${PATH}"
export PYENV_SHELL=python3.7
command pyenv rehash 2>/dev/null
pyenv() {
  local command
  command="${1:-}"
  if [ "$#" -gt 0 ]; then
    shift
  fi

  case "$command" in
  activate|deactivate|rehash|shell)
    eval "$(pyenv "sh-$command" "$@")";;
  *)
    command pyenv "$command" "$@";;
  esac
}
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv virtualenv-init -
export PATH="/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
  local ret=$?
  if [ -n "$VIRTUAL_ENV" ]; then
    eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
  else
    eval "$(pyenv sh-activate --quiet || true)" || true
  fi
  return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
  PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi

Ok, so if we can load these as bash, we can get all the commands correctly into xonsh. Fortunately there is a command in xonsh just for this:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash --help
usage: source-foreign [-h] [-i INTERACTIVE] [-l LOGIN] [--envcmd ENVCMD]
                      [--aliascmd ALIASCMD] [--extra-args EXTRA_ARGS]
                      [-s SAFE] [-p PREVCMD] [--postcmd POSTCMD]
                      [--funcscmd FUNCSCMD] [--sourcer SOURCER]
                      [--use-tmpfile USE_TMPFILE]
                      [--seterrprevcmd SETERRPREVCMD]
                      [--seterrpostcmd SETERRPOSTCMD] [--overwrite-aliases]
                      [--suppress-skip-message] [--show] [-d]
                      shell files_or_code [files_or_code ...]

Sources a file written in a foreign shell language.

positional arguments:
  shell                 Name or path to the foreign shell
  files_or_code         file paths to source or code in the target language.
...

jtuckey@jay-x1-carbon ~ <xonsh>$ aliases
xonsh.aliases.Aliases(
{...
 'source-bash': ['source-foreign', 'bash', '--sourcer=source']
...}

So we should be able to just source the output of the pyenv commands. Let’s try it:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
# Shell just hangs at this point....

Sooo…. that’s not working. My shell just locks up. I can’t even Ctrl-C to kill it.

What’s going on here? After a bit of investigation, what seems to be happening is this: when you use source-bash the bash shell launched to run the source is running the .bashrc file, which puts it back into xonsh, where it gets stuck. This makes sense, although it’s hard to work out. To test this, lets get rid of the xonsh lines from .bashrc and see if it works correctly then.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
#if xonsh; then
#  exit
#fi

And try running it manually

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ 

Looking better. Now I’m back where I started, though, so how can I run xonsh as my shell. From the xonsh tutorial there is this line:

Alternatively, you can setup your terminal emulator (xterm, gnome-terminal, etc) to run xonsh automatically when it starts up. This is recommended.

However, in my gnome-terminal if I try setting xonsh as the command it can’t find xonsh. This is because the python 3.7 environment where xonsh lives is initialised during bash startup, not gui startup, so my gnome-terminal can’t find the xonsh command.

Ok, so how do I put something into my whole user environment? That’s what the .profile file in your user profile is for. Let’s put the pyenv init stuff into the end of my .profile file:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Now with a logoff and back on, let’s see what happens. Now when I set gnome-terminal to run xonsh, it works. However:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.


jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv virtualenv-init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Hmmm, still not quite there, but closer.

When I heard about the xonsh project – https://xon.sh I thought it sounded great, as I really enjoy the python language and am much more comfortable using python to write scripts than I am using bash. I also find the bash syntax very confusing, and am never sure of the right way to write a script or use a variable.

So, I decided to install it and give it a try! When using python on Linux I always use the pyenv tool to install and manage my python versions, as well as my virtualenvs. This tool allows me to easily install the latest version of python, as well as create a virtualenv for a specific project and keep all the packages that project uses separate from each other and my system python. It’s a great tool, and I can highly recommend using it.

pyenv installs all python versions into your user profile, and allows you to select from them on the command line using

pyenv shell 3.7.0

To get all the pyenv commands, along with the pyenv command completion (which is very good), you add an init command to the end of your .bashrc file. This is how it looks in mine:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Excellent! When running in bash this sets up pyenv, then sets my shell to use python 3.7.0. So let’s install xonsh:

jtuckey@jay-x1-carbon:~$ pip install --user xonsh
Collecting xonsh
  Using cached https://files.pythonhosted.org/packages/58/16/fce8ecc880a44dfcb06b22f638df0b6982ad588eb0f2080fbd5218a42340/xonsh-0.8.0.tar.gz
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.8.0

Looking good so far! I can now use the xonsh command to hop into a xonsh shell, and then use all the goodness of a python-based shell:

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ $ ${'XONSH_VERSION'}
'0.8.0'
jtuckey@jay-x1-carbon ~ $ history info
backend: json
sessionid: 30e7e6d4-4945-4f99-9b65-d38a2ab61da2
filename: /home/jtuckey/.local/share/xonsh/xonsh-30e7e6d4-4945-4f99-9b65-d38a2ab61da2.json
length: 1
buffersize: 100
bufferlength: 1
gc options: (8128, 'commands')

Now that I have it installed and am getting all the good features, I naturally wanted it as my default shell. My first attempt at this was to simply put it the end of my .bashrc file:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

xonsh

This works! However, to quit the shell, I need to exit twice:

jtuckey@jay-x1-carbon ~ <xonsh>$ exit
jtuckey@jay-x1-carbon:~$ exit

Ok, maybe let’s put it in an auto-exiting block. Back to the .bashrc:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
if xonsh; then
  exit
fi

This works even better. Now it will automatically exit when I’m done using xonsh, but if it can’t find xonsh it will stay in bash. However, with this setup, I try to do some work, which involves activating one of my pyenv virtualenvs. This gives me an error:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Ok, so clearly the pyenv setup chunk from bash isn’t carrying across nicely from bash to xonsh. The path looks alright:

jtuckey@jay-x1-carbon ~ <xonsh>$ $PATH
\EnvPath(
['/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.local/bin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']
)

Let’s have a look at what those two pyenv init commands. What they are doing is generating a bit of code using pyenv init - pyenv virtualenv-init - and then evaluating it in the local scope. We can easily see the code that’s being generated:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv init -
export PATH="/home/jtuckey/.pyenv/shims:${PATH}"
export PYENV_SHELL=python3.7
command pyenv rehash 2>/dev/null
pyenv() {
  local command
  command="${1:-}"
  if [ "$#" -gt 0 ]; then
    shift
  fi

  case "$command" in
  activate|deactivate|rehash|shell)
    eval "$(pyenv "sh-$command" "$@")";;
  *)
    command pyenv "$command" "$@";;
  esac
}
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv virtualenv-init -
export PATH="/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
  local ret=$?
  if [ -n "$VIRTUAL_ENV" ]; then
    eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
  else
    eval "$(pyenv sh-activate --quiet || true)" || true
  fi
  return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
  PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi

Ok, so if we can load these as bash, we can get all the commands correctly into xonsh. Fortunately there is a command in xonsh just for this:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash --help
usage: source-foreign [-h] [-i INTERACTIVE] [-l LOGIN] [--envcmd ENVCMD]
                      [--aliascmd ALIASCMD] [--extra-args EXTRA_ARGS]
                      [-s SAFE] [-p PREVCMD] [--postcmd POSTCMD]
                      [--funcscmd FUNCSCMD] [--sourcer SOURCER]
                      [--use-tmpfile USE_TMPFILE]
                      [--seterrprevcmd SETERRPREVCMD]
                      [--seterrpostcmd SETERRPOSTCMD] [--overwrite-aliases]
                      [--suppress-skip-message] [--show] [-d]
                      shell files_or_code [files_or_code ...]

Sources a file written in a foreign shell language.

positional arguments:
  shell                 Name or path to the foreign shell
  files_or_code         file paths to source or code in the target language.
...

jtuckey@jay-x1-carbon ~ <xonsh>$ aliases
xonsh.aliases.Aliases(
{...
 'source-bash': ['source-foreign', 'bash', '--sourcer=source']
...}

So we should be able to just source the output of the pyenv commands. Let’s try it:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
# Shell just hangs at this point....

Sooo…. that’s not working. My shell just locks up. I can’t even Ctrl-C to kill it.

What’s going on here? After a bit of investigation, what seems to be happening is this: when you use source-bash the bash shell launched to run the source is running the .bashrc file, which puts it back into xonsh, where it gets stuck. This makes sense, although it’s hard to work out. To test this, lets get rid of the xonsh lines from .bashrc and see if it works correctly then.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
#if xonsh; then
#  exit
#fi

And try running it manually

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ 

Looking better. Now I’m back where I started, though, so how can I run xonsh as my shell. From the xonsh tutorial there is this line:

Alternatively, you can setup your terminal emulator (xterm, gnome-terminal, etc) to run xonsh automatically when it starts up. This is recommended.

However, in my gnome-terminal if I try setting xonsh as the command it can’t find xonsh. This is because the python 3.7 environment where xonsh lives is initialised during bash startup, not gui startup, so my gnome-terminal can’t find the xonsh command.

Ok, so how do I put something into my whole user environment? That’s what the .profile file in your user profile is for. Let’s put the pyenv init stuff into the end of my .profile file:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Now with a logoff and back on, let’s see what happens. Now when I set gnome-terminal to run xonsh, it works. However:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.


jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv virtualenv-init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Hmmm, still not quite there, but closer. After some further research, I found this on a page about virtual environments and xonsh.

The usual tools for creating Python virtual environments—venv, virtualenv, pew—don’t play well with xonsh. …. Luckily, xonsh ships with its own virtual environments manager called Vox.

https://xon.sh/python_virtual_environments.html

Sounds like vox is the way to go. To load vox you just need to load it. However, by default it doesn’t look for virtualenvs in the pyenv path. This is easily fixable by setting the $VIRTUALENV_HOME variable:

jay@jay-alienware-ubuntu:~:x$ xontrib load vox                                                                               
jay@jay-alienware-ubuntu:~:x$ vox list                                                                                       
No environments available. Create one with "vox new".

jay@jay-alienware-ubuntu:~:x$ $VIRTUALENV_HOME = $HOME +'/.pyenv/versions'                                                   
jay@jay-alienware-ubuntu:~:x$ vox list                                                                                       
Available environments:
3.6.2/envs/test-install
3.6.2/envs/tests
3.6.4/envs/mdb
3.7.0/envs/xonshdev
jay@jay-alienware-ubuntu:~:x$ vox activate tests                                                                             
Activated "tests".

(tests) jay@jay-alienware-ubuntu:~:x$ vox deactivate                                                                         
Deactivated "tests".

jay@jay-alienware-ubuntu:~:x$  

Be aware that creating a virtualenv through vox will just use your currently set python version from pyenv. However, you can still use pyenv to create a new virtualenv of any version, and then use vox to activate it:

jay@jay-alienware-ubuntu:~:x$ pyenv virtualenv 3.6.4 newenv                                                                  
Requirement already satisfied: setuptools in /home/jay/.pyenv/versions/3.6.4/envs/newenv/lib/python3.6/site-packages
Requirement already satisfied: pip in /home/jay/.pyenv/versions/3.6.4/envs/newenv/lib/python3.6/site-packages
jay@jay-alienware-ubuntu:~:x$ vox activate newenv                                                                            
Activated "newenv".

(newenv) jay@jay-alienware-ubuntu:~:x$  

So I now have a fully working pyenv+xonsh setup! The only thing that’s missing is proper tab-completion of the pyenv commands, which is something I may look into in the future, but for now I’m able to do my work easily without leaving the xonsh shell, which is great!