Python@2 complains: 'import sitecustomize' failed; use -v for traceback


(Sun Qingyao) #1

I didn’t set PYTHONPATH for python2/3, but python@2 still complains when starting up

$echo $PYTHONPATH


$ python2
'import sitecustomize' failed; use -v for traceback
Python 2.7.14 (default, Mar  9 2018, 23:57:12)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

More verbose version here:

$ python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py
import site # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py
import os # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py
import posixpath # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.py
import stat # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.py
import genericpath # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.py
import warnings # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.py
import linecache # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.py
import types # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py
import UserDict # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.py
import abc # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.py
import traceback # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py
import sysconfig # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py
import re # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.pyc
import _sre # builtin
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.pyc
dlopen("/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so", 2);
import _locale # dynamically loaded from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.py
import _osx_support # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.pyc
# /usr/local/lib/python3.6/site-packages/sitecustomize.pyc matches /usr/local/lib/python3.6/site-packages/sitecustomize.py
import sitecustomize # precompiled from /usr/local/lib/python3.6/site-packages/sitecustomize.pyc
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py", line 489, in execsitecustomize
    import sitecustomize
  File "/usr/local/lib/python3.6/site-packages/sitecustomize.py", line 15, in <module>
    str(sys.version_info[0]) + '.x!\n     PYTHONPATH is currently: "' + str(os.environ['PYTHONPATH']) + '"\n' +
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'PYTHONPATH'
import encodings # directory /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.py
import encodings # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.py
import codecs # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.pyc
import _codecs # builtin
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.pyc
# /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc matches /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc
Python 2.7.14 (default, Mar  9 2018, 23:57:12)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so", 2);
import readline # dynamically loaded from /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so
>>>

Here is the relevant part from the offending file (/usr/local/lib/python3.6/site-packages/sitecustomize.py):

if sys.version_info[0] != 3:
    # This can only happen if the user has set the PYTHONPATH for 3.x and run Python 2.x or vice versa.
    # Every Python looks at the PYTHONPATH variable and we can't fix it here in sitecustomize.py,
    # because the PYTHONPATH is evaluated after the sitecustomize.py. Many modules (e.g. PyQt4) are
    # built only for a specific version of Python and will fail with cryptic error messages.
    # In the end this means: Don't set the PYTHONPATH permanently if you use different Python versions.
    exit('Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python ' +
         str(sys.version_info[0]) + '.x!\n     PYTHONPATH is currently: "' + str(os.environ['PYTHONPATH']) + '"\n' +
         '     You should `unset PYTHONPATH` to fix this.')

I’ve run multiple times of brew update && brew upgrade, and here is the output of brew doctor, though I cannot see anything related to Python:

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/libcmocka.0.4.1.dylib
  /usr/local/lib/libtcl8.6.dylib
  /usr/local/lib/libtk8.6.dylib

Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected header files:
  /usr/local/include/cmocka.h
  /usr/local/include/cmocka_pbc.h
  /usr/local/include/fakemysql.h
  /usr/local/include/fakepq.h
  /usr/local/include/fakesql.h
  /usr/local/include/itcl.h
  /usr/local/include/itcl2TclOO.h
  /usr/local/include/itclDecls.h
  /usr/local/include/itclInt.h
  /usr/local/include/itclIntDecls.h
  /usr/local/include/itclMigrate2TclCore.h
  /usr/local/include/itclTclIntStubsFcn.h
  /usr/local/include/mysqlStubs.h
  /usr/local/include/odbcStubs.h
  /usr/local/include/pqStubs.h
  /usr/local/include/tcl.h
  /usr/local/include/tclDecls.h
  /usr/local/include/tclOO.h
  /usr/local/include/tclOODecls.h
  /usr/local/include/tclPlatDecls.h
  /usr/local/include/tclThread.h
  /usr/local/include/tclTomMath.h
  /usr/local/include/tclTomMathDecls.h
  /usr/local/include/tdbc.h
  /usr/local/include/tdbcDecls.h
  /usr/local/include/tdbcInt.h
  /usr/local/include/tk.h
  /usr/local/include/tkDecls.h
  /usr/local/include/tkPlatDecls.h

Warning: Unbrewed .pc files were found in /usr/local/lib/pkgconfig.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .pc files:
  /usr/local/lib/pkgconfig/cmocka.pc
  /usr/local/lib/pkgconfig/tcl.pc
  /usr/local/lib/pkgconfig/tk.pc

Warning: Unbrewed static libraries were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected static libraries:
  /usr/local/lib/libcmocka.a
  /usr/local/lib/libtclstub8.6.a
  /usr/local/lib/libtkstub8.6.a

(Wu Yongwei) #2

Don’t you feel weird that your Python 2.7 installation found a file under Python 3.6? I would suggest you fully uninstall python and python@2, and then reinstall them as needed (python2 requires python@2).


(Sun Qingyao) #3

Did a brew reinstall python python2 python@2, but the problem remains :frowning:


(Inada Naoki) #4

Maybe, there are some wrong pth files somewhere. Try this for clean install:

brew uninstall python@2 python
rm -rf /usr/local/lib/python*
brew install python python@2

(But I recommend you to use Python 3 only, and remove python@2…)


(Robert Nsinga) #5

Hey @sunqingyao, I found a homebrew.pth inside the default Python2 site-packages for MacOSX (High Sierra) ~/Library/Python/2.7/lib/python/site-packages. Here is the file’s content:

import sys; sys.path.insert(1, "/usr/local/lib/python2.7/site-packages")
import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")
import sys; sys.path.insert(1, "/usr/local/lib/python3.6/site-packages")

All I did is remove that last line. That’s it. Maybe the trick could work for Homebrew’s Python@2 :smiley: :sunglasses:


(Wu Yongwei) #6

That’s weird. I do not have the path /Library/Python/2.7/lib at all (also High Sierra). And I would further suggest that the system python and Homebrew python should be separated.


(Robert Nsinga) #7

Oops! I meant ~/Library/Python/2.7/lib. It’s in the user’s files, not system files.


(Alex Reinisch) #8

Thanks for all your investigation! I agree, the culprit seems to be the file ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth.

If anyone else comes this way and is looking for a “scorched-earth,” full solution, this is what worked for me:

brew uninstall --ignore-dependencies python python3 python@2  # Get rid of all homebrew pythons.
brew uninstall --force python  # Really.
rm -rf /usr/local/lib/python*  # Remove homebrew python libraries.
rm -rf ~/Library/Python/  # Remove any local complications homebrew may have thrown in.
brew install python python@2  # Install pythons.

# Done here.  Optionally continue to prove to yourself that all is well:
pip install --upgrade pip setuptools
pip3 install --upgrade pip setuptools wheel
python -V   # Python 2.7.14
python3 -V  # Python 3.6.5
pip -V      # pip 10.0.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)
pip3 -V     # pip 10.0.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

(Patrick Cudahy) #9

I’m having the same issue with linuxbrew on ubuntu. Any idea what the analog of ~/Library/Python would be on linux?


(Shaun Jackman) #10

To print the default search path of Python print…

python3 -c 'import sys; print(sys.path)' | sed 's/, /,\n/g'

On Python 3, the default user site-packages directory is…

~/.local/lib/python3.7/site-packages

See https://www.python.org/dev/peps/pep-0370/

As far as I can tell, Python 2 does not have a default user site-packages directory.


(Patrick Cudahy) #11

Thanks for the detailed response. A quick rm -rf ~/.local/lib/python* along with the steps above from @areinisc seems to have solved my problem.