Author: Jay Tuckey

When I heard about the xonsh project – https://xon.sh I thought it sounded great, as I really enjoy the python language and am much more comfortable using python to write scripts than I am using bash. I also find the bash syntax very confusing, and am never sure of the right way to write a script or use a variable.

So, I decided to install it and give it a try! When using python on Linux I always use the pyenv tool to install and manage my python versions, as well as my virtualenvs. This tool allows me to easily install the latest version of python, as well as create a virtualenv for a specific project and keep all the packages that project uses separate from each other and my system python. It’s a great tool, and I can highly recommend using it.

pyenv installs all python versions into your user profile, and allows you to select from them on the command line using

pyenv shell 3.7.0

To get all the pyenv commands, along with the pyenv command completion (which is very good), you add an init command to the end of your .bashrc file. This is how it looks in mine:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Excellent! When running in bash this sets up pyenv, then sets my shell to use python 3.7.0. So let’s install xonsh:

jtuckey@jay-x1-carbon:~$ pip install --user xonsh
Collecting xonsh
  Using cached https://files.pythonhosted.org/packages/58/16/fce8ecc880a44dfcb06b22f638df0b6982ad588eb0f2080fbd5218a42340/xonsh-0.8.0.tar.gz
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.8.0

Looking good so far! I can now use the xonsh command to hop into a xonsh shell, and then use all the goodness of a python-based shell:

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ $ ${'XONSH_VERSION'}
'0.8.0'
jtuckey@jay-x1-carbon ~ $ history info
backend: json
sessionid: 30e7e6d4-4945-4f99-9b65-d38a2ab61da2
filename: /home/jtuckey/.local/share/xonsh/xonsh-30e7e6d4-4945-4f99-9b65-d38a2ab61da2.json
length: 1
buffersize: 100
bufferlength: 1
gc options: (8128, 'commands')

Now that I have it installed and am getting all the good features, I naturally wanted it as my default shell. My first attempt at this was to simply put it the end of my .bashrc file:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

xonsh

This works! However, to quit the shell, I need to exit twice:

jtuckey@jay-x1-carbon ~ <xonsh>$ exit
jtuckey@jay-x1-carbon:~$ exit

Ok, maybe let’s put it in an auto-exiting block. Back to the .bashrc:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
if xonsh; then
  exit
fi

This works even better. Now it will automatically exit when I’m done using xonsh, but if it can’t find xonsh it will stay in bash. However, with this setup, I try to do some work, which involves activating one of my pyenv virtualenvs. This gives me an error:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Ok, so clearly the pyenv setup chunk from bash isn’t carrying across nicely from bash to xonsh. The path looks alright:

jtuckey@jay-x1-carbon ~ <xonsh>$ $PATH
\EnvPath(
['/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.local/bin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']
)

Let’s have a look at what those two pyenv init commands. What they are doing is generating a bit of code using pyenv init - pyenv virtualenv-init - and then evaluating it in the local scope. We can easily see the code that’s being generated:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv init -
export PATH="/home/jtuckey/.pyenv/shims:${PATH}"
export PYENV_SHELL=python3.7
command pyenv rehash 2>/dev/null
pyenv() {
  local command
  command="${1:-}"
  if [ "$#" -gt 0 ]; then
    shift
  fi

  case "$command" in
  activate|deactivate|rehash|shell)
    eval "$(pyenv "sh-$command" "$@")";;
  *)
    command pyenv "$command" "$@";;
  esac
}
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv virtualenv-init -
export PATH="/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
  local ret=$?
  if [ -n "$VIRTUAL_ENV" ]; then
    eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
  else
    eval "$(pyenv sh-activate --quiet || true)" || true
  fi
  return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
  PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi

Ok, so if we can load these as bash, we can get all the commands correctly into xonsh. Fortunately there is a command in xonsh just for this:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash --help
usage: source-foreign [-h] [-i INTERACTIVE] [-l LOGIN] [--envcmd ENVCMD]
                      [--aliascmd ALIASCMD] [--extra-args EXTRA_ARGS]
                      [-s SAFE] [-p PREVCMD] [--postcmd POSTCMD]
                      [--funcscmd FUNCSCMD] [--sourcer SOURCER]
                      [--use-tmpfile USE_TMPFILE]
                      [--seterrprevcmd SETERRPREVCMD]
                      [--seterrpostcmd SETERRPOSTCMD] [--overwrite-aliases]
                      [--suppress-skip-message] [--show] [-d]
                      shell files_or_code [files_or_code ...]

Sources a file written in a foreign shell language.

positional arguments:
  shell                 Name or path to the foreign shell
  files_or_code         file paths to source or code in the target language.
...

jtuckey@jay-x1-carbon ~ <xonsh>$ aliases
xonsh.aliases.Aliases(
{...
 'source-bash': ['source-foreign', 'bash', '--sourcer=source']
...}

So we should be able to just source the output of the pyenv commands. Let’s try it:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
# Shell just hangs at this point....

Sooo…. that’s not working. My shell just locks up. I can’t even Ctrl-C to kill it.

What’s going on here? After a bit of investigation, what seems to be happening is this: when you use source-bash the bash shell launched to run the source is running the .bashrc file, which puts it back into xonsh, where it gets stuck. This makes sense, although it’s hard to work out. To test this, lets get rid of the xonsh lines from .bashrc and see if it works correctly then.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
#if xonsh; then
#  exit
#fi

And try running it manually

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ 

Looking better. Now I’m back where I started, though, so how can I run xonsh as my shell. From the xonsh tutorial there is this line:

Alternatively, you can setup your terminal emulator (xterm, gnome-terminal, etc) to run xonsh automatically when it starts up. This is recommended.

However, in my gnome-terminal if I try setting xonsh as the command it can’t find xonsh. This is because the python 3.7 environment where xonsh lives is initialised during bash startup, not gui startup, so my gnome-terminal can’t find the xonsh command.

Ok, so how do I put something into my whole user environment? That’s what the .profile file in your user profile is for. Let’s put the pyenv init stuff into the end of my .profile file:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Now with a logoff and back on, let’s see what happens. Now when I set gnome-terminal to run xonsh, it works. However:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.


jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv virtualenv-init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Hmmm, still not quite there, but closer.

When I heard about the xonsh project – https://xon.sh I thought it sounded great, as I really enjoy the python language and am much more comfortable using python to write scripts than I am using bash. I also find the bash syntax very confusing, and am never sure of the right way to write a script or use a variable.

So, I decided to install it and give it a try! When using python on Linux I always use the pyenv tool to install and manage my python versions, as well as my virtualenvs. This tool allows me to easily install the latest version of python, as well as create a virtualenv for a specific project and keep all the packages that project uses separate from each other and my system python. It’s a great tool, and I can highly recommend using it.

pyenv installs all python versions into your user profile, and allows you to select from them on the command line using

pyenv shell 3.7.0

To get all the pyenv commands, along with the pyenv command completion (which is very good), you add an init command to the end of your .bashrc file. This is how it looks in mine:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Excellent! When running in bash this sets up pyenv, then sets my shell to use python 3.7.0. So let’s install xonsh:

jtuckey@jay-x1-carbon:~$ pip install --user xonsh
Collecting xonsh
  Using cached https://files.pythonhosted.org/packages/58/16/fce8ecc880a44dfcb06b22f638df0b6982ad588eb0f2080fbd5218a42340/xonsh-0.8.0.tar.gz
Installing collected packages: xonsh
  Running setup.py install for xonsh ... done
Successfully installed xonsh-0.8.0

Looking good so far! I can now use the xonsh command to hop into a xonsh shell, and then use all the goodness of a python-based shell:

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ $ ${'XONSH_VERSION'}
'0.8.0'
jtuckey@jay-x1-carbon ~ $ history info
backend: json
sessionid: 30e7e6d4-4945-4f99-9b65-d38a2ab61da2
filename: /home/jtuckey/.local/share/xonsh/xonsh-30e7e6d4-4945-4f99-9b65-d38a2ab61da2.json
length: 1
buffersize: 100
bufferlength: 1
gc options: (8128, 'commands')

Now that I have it installed and am getting all the good features, I naturally wanted it as my default shell. My first attempt at this was to simply put it the end of my .bashrc file:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

xonsh

This works! However, to quit the shell, I need to exit twice:

jtuckey@jay-x1-carbon ~ <xonsh>$ exit
jtuckey@jay-x1-carbon:~$ exit

Ok, maybe let’s put it in an auto-exiting block. Back to the .bashrc:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
if xonsh; then
  exit
fi

This works even better. Now it will automatically exit when I’m done using xonsh, but if it can’t find xonsh it will stay in bash. However, with this setup, I try to do some work, which involves activating one of my pyenv virtualenvs. This gives me an error:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Ok, so clearly the pyenv setup chunk from bash isn’t carrying across nicely from bash to xonsh. The path looks alright:

jtuckey@jay-x1-carbon ~ <xonsh>$ $PATH
\EnvPath(
['/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims',
 '/home/jtuckey/.pyenv/shims',
 '/home/jtuckey/.pyenv/bin',
 '/home/jtuckey/.local/bin',
 '/usr/local/sbin',
 '/usr/local/bin',
 '/usr/sbin',
 '/usr/bin',
 '/sbin',
 '/bin',
 '/usr/games',
 '/usr/local/games',
 '/snap/bin']
)

Let’s have a look at what those two pyenv init commands. What they are doing is generating a bit of code using pyenv init - pyenv virtualenv-init - and then evaluating it in the local scope. We can easily see the code that’s being generated:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv init -
export PATH="/home/jtuckey/.pyenv/shims:${PATH}"
export PYENV_SHELL=python3.7
command pyenv rehash 2>/dev/null
pyenv() {
  local command
  command="${1:-}"
  if [ "$#" -gt 0 ]; then
    shift
  fi

  case "$command" in
  activate|deactivate|rehash|shell)
    eval "$(pyenv "sh-$command" "$@")";;
  *)
    command pyenv "$command" "$@";;
  esac
}
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv virtualenv-init -
export PATH="/home/jtuckey/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
  local ret=$?
  if [ -n "$VIRTUAL_ENV" ]; then
    eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
  else
    eval "$(pyenv sh-activate --quiet || true)" || true
  fi
  return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
  PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi

Ok, so if we can load these as bash, we can get all the commands correctly into xonsh. Fortunately there is a command in xonsh just for this:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash --help
usage: source-foreign [-h] [-i INTERACTIVE] [-l LOGIN] [--envcmd ENVCMD]
                      [--aliascmd ALIASCMD] [--extra-args EXTRA_ARGS]
                      [-s SAFE] [-p PREVCMD] [--postcmd POSTCMD]
                      [--funcscmd FUNCSCMD] [--sourcer SOURCER]
                      [--use-tmpfile USE_TMPFILE]
                      [--seterrprevcmd SETERRPREVCMD]
                      [--seterrpostcmd SETERRPOSTCMD] [--overwrite-aliases]
                      [--suppress-skip-message] [--show] [-d]
                      shell files_or_code [files_or_code ...]

Sources a file written in a foreign shell language.

positional arguments:
  shell                 Name or path to the foreign shell
  files_or_code         file paths to source or code in the target language.
...

jtuckey@jay-x1-carbon ~ <xonsh>$ aliases
xonsh.aliases.Aliases(
{...
 'source-bash': ['source-foreign', 'bash', '--sourcer=source']
...}

So we should be able to just source the output of the pyenv commands. Let’s try it:

jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
# Shell just hangs at this point....

Sooo…. that’s not working. My shell just locks up. I can’t even Ctrl-C to kill it.

What’s going on here? After a bit of investigation, what seems to be happening is this: when you use source-bash the bash shell launched to run the source is running the .bashrc file, which puts it back into xonsh, where it gets stuck. This makes sense, although it’s hard to work out. To test this, lets get rid of the xonsh lines from .bashrc and see if it works correctly then.

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

# Start xonsh shell
#if xonsh; then
#  exit
#fi

And try running it manually

jtuckey@jay-x1-carbon:~$ xonsh
jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ 

Looking better. Now I’m back where I started, though, so how can I run xonsh as my shell. From the xonsh tutorial there is this line:

Alternatively, you can setup your terminal emulator (xterm, gnome-terminal, etc) to run xonsh automatically when it starts up. This is recommended.

However, in my gnome-terminal if I try setting xonsh as the command it can’t find xonsh. This is because the python 3.7 environment where xonsh lives is initialised during bash startup, not gui startup, so my gnome-terminal can’t find the xonsh command.

Ok, so how do I put something into my whole user environment? That’s what the .profile file in your user profile is for. Let’s put the pyenv init stuff into the end of my .profile file:

# Load pyenv automatically by adding
# the following to ~/.bash_profile:

export PATH="/home/jtuckey/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv shell 3.7.0

Now with a logoff and back on, let’s see what happens. Now when I set gnome-terminal to run xonsh, it works. However:

jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.


jtuckey@jay-x1-carbon ~ <xonsh>$ source-bash $(pyenv virtualenv-init - bash)
Skipping application of 'll' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
Skipping application of 'ls' alias from 'bash' since it shares a name with an existing xonsh alias. Use "--overwrite-alias" option to apply it anyway.You may prevent this message with "--suppress-skip-message" or "$FOREIGN_ALIASES_SUPPRESS_SKIP_MESSAGE = True".
jtuckey@jay-x1-carbon ~ <xonsh>$ pyenv activate godev

Failed to activate virtualenv.

Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.

Hmmm, still not quite there, but closer. After some further research, I found this on a page about virtual environments and xonsh.

The usual tools for creating Python virtual environments—venv, virtualenv, pew—don’t play well with xonsh. …. Luckily, xonsh ships with its own virtual environments manager called Vox.

https://xon.sh/python_virtual_environments.html

Sounds like vox is the way to go. To load vox you just need to load it. However, by default it doesn’t look for virtualenvs in the pyenv path. This is easily fixable by setting the $VIRTUALENV_HOME variable:

jay@jay-alienware-ubuntu:~:x$ xontrib load vox                                                                               
jay@jay-alienware-ubuntu:~:x$ vox list                                                                                       
No environments available. Create one with "vox new".

jay@jay-alienware-ubuntu:~:x$ $VIRTUALENV_HOME = $HOME +'/.pyenv/versions'                                                   
jay@jay-alienware-ubuntu:~:x$ vox list                                                                                       
Available environments:
3.6.2/envs/test-install
3.6.2/envs/tests
3.6.4/envs/mdb
3.7.0/envs/xonshdev
jay@jay-alienware-ubuntu:~:x$ vox activate tests                                                                             
Activated "tests".

(tests) jay@jay-alienware-ubuntu:~:x$ vox deactivate                                                                         
Deactivated "tests".

jay@jay-alienware-ubuntu:~:x$  

Be aware that creating a virtualenv through vox will just use your currently set python version from pyenv. However, you can still use pyenv to create a new virtualenv of any version, and then use vox to activate it:

jay@jay-alienware-ubuntu:~:x$ pyenv virtualenv 3.6.4 newenv                                                                  
Requirement already satisfied: setuptools in /home/jay/.pyenv/versions/3.6.4/envs/newenv/lib/python3.6/site-packages
Requirement already satisfied: pip in /home/jay/.pyenv/versions/3.6.4/envs/newenv/lib/python3.6/site-packages
jay@jay-alienware-ubuntu:~:x$ vox activate newenv                                                                            
Activated "newenv".

(newenv) jay@jay-alienware-ubuntu:~:x$  

So I now have a fully working pyenv+xonsh setup! The only thing that’s missing is proper tab-completion of the pyenv commands, which is something I may look into in the future, but for now I’m able to do my work easily without leaving the xonsh shell, which is great!