I am facing the following error on Jupyter Notebook when attempting to import fancy impute please help
EDIT:
Machine details: Core i5-4200u, 8GB DDR3 RAM, NVIDIA GT750M Graphics
Environment Details: Running the latest Anaconda version, using Jupyter Notebook from inside the Anaconda application.
I am not sure if I should have attached the entire code. However, prior to running the import I had run pip install multiple times to upgrade numpy, keras, ecos etc as per suggestions available on StackOverflow etc.
import fancyimpute
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-3f82472c4160> in <module>
----> 1 import fancyimpute
~\AppData\Roaming\Python\Python38\site-packages\fancyimpute\__init__.py in <module>
2
3 from .solver import Solver
----> 4 from .nuclear_norm_minimization import NuclearNormMinimization
5 from .matrix_factorization import MatrixFactorization
6 from .iterative_svd import IterativeSVD
~\AppData\Roaming\Python\Python38\site-packages\fancyimpute\nuclear_norm_minimization.py in <module>
11 # limitations under the License.
12
---> 13 import cvxpy
14
15 from .solver import Solver
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\__init__.py in <module>
16
17 __version__ = "1.1.13"
---> 18 from cvxpy.atoms import *
19 from cvxpy.constraints import NonPos, Zero, SOC, PSD
20 from cvxpy.expressions.expression import Expression
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\__init__.py in <module>
18 from cvxpy.atoms.dist_ratio import dist_ratio
19 from cvxpy.atoms.eye_minus_inv import eye_minus_inv, resolvent
---> 20 from cvxpy.atoms.geo_mean import geo_mean
21 from cvxpy.atoms.gen_lambda_max import gen_lambda_max
22 from cvxpy.atoms.gmatmul import gmatmul
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\geo_mean.py in <module>
18 import numpy as np
19 import scipy.sparse as sp
---> 20 from cvxpy.utilities.power_tools import (fracify, decompose, approx_error, lower_bound,
21 over_bound, prettydict)
22 from typing import List, Optional, Tuple
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\utilities\power_tools.py in <module>
16
17 from fractions import Fraction
---> 18 from cvxpy.atoms.affine.reshape import reshape
19 from cvxpy.atoms.affine.vstack import vstack
20 from cvxpy.constraints.second_order import SOC
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\affine\reshape.py in <module>
16
17 from cvxpy.expressions.expression import Expression
---> 18 from cvxpy.atoms.affine.hstack import hstack
19 from cvxpy.atoms.affine.affine_atom import AffAtom
20 from typing import Tuple
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\affine\hstack.py in <module>
17
18 import cvxpy.lin_ops.lin_utils as lu
---> 19 from cvxpy.atoms.affine.affine_atom import AffAtom
20 import numpy as np
21
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\affine\affine_atom.py in <module>
20 import cvxpy.lin_ops.lin_utils as lu
21 from cvxpy.atoms.atom import Atom
---> 22 from cvxpy.cvxcore.python import canonInterface
23 from cvxpy.expressions.constants import Constant
24 from cvxpy.utilities import performance_utils as perf
~\AppData\Roaming\Python\Python38\site-packages\cvxpy\cvxcore\python\__init__.py in <module>
1 # TODO(akshayka): This is a hack; the swig-auto-generated cvxcore.py
2 # tries to import cvxcore as `from . import _cvxcore`
----> 3 import _cvxcore
ImportError: numpy.core.multiarray failed to import
The error:
Kenobi, we now have a more elegant method of posting traceback errors for a more civilized age of asking for help.
Please donāt use screenshots. And please donāt post a fraction of the information as it makes it harder for those in the community to help you.
For posting the traceback as copied text learn about posting code blocks between triple back ticks, see the āBlock Codeā section here, in particular look at how to do it below the text that says āAnother (short) example, to demonstrate how to wrap your code blocksā. Discourse makes this easy; however, if you feel the traceback is long you can post it in a gist or a code snippet site and link to it in your post as well.
Also please review Getting good answers to your questions.
You donāt describe how you installed Jupyter notebook? How you installed cvxpy? Conda or pip? What type of machine you are on? Etc.?
Testable code examples would also be very nice to include in the future. That way we can test things are working in a fresh environment. They donāt have to be your code even, just related examples that involve imports of the same items.
That being said, my guess based on the limited information provided seems to be incompatibility with packages in your environment. Based on an open similar issue in the Issues listing for cvxpy and forcing an update, Iād suggest starting by installing cvxpy again by running the following at the following in a new cell right near the top of your notebook:
%pip install --upgrade --force-reinstall cvxpy
Following running that code, restart the kernel for the notebook, and then try running your code again.
If that simple approach fails, based on a very similar issue on StackOveflow here try to make more of the packages consistent. Maybe something like this in a new cell at the top:
%pip install --upgrade --force-reinstall numpy
%pip install --upgrade --force-reinstall scipy
%pip install --upgrade --force-reinstall cvxpy
%pip install --upgrade --force-reinstall fancyimpute
The fresh installs will hopefully make everything current and compatible in the notebookās environment.
1 Like
I ran the following trouble shoots
conda update --all
conda create -n py36 python=3.6 anaconda
conda activate
conda install ecos
conda install CVXcanon
conda install -c cvxgrp scs
conda install scs
pip install fancyimpute
after this I ran
import fancyimpute
Now I am getting the following error:
ImportError Traceback (most recent call last)
<ipython-input-1-3f82472c4160> in <module>
----> 1 import fancyimpute
C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\fancyimpute\__init__.py in <module>
3 from .solver import Solver
4 from .nuclear_norm_minimization import NuclearNormMinimization
----> 5 from .matrix_factorization import MatrixFactorization
6 from .iterative_svd import IterativeSVD
7 from .simple_fill import SimpleFill
C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\fancyimpute\matrix_factorization.py in <module>
20 from .common import import_from
21 from .scaler import Scaler
---> 22 from .keras_models import KerasMatrixFactorizer
23 from .solver import Solver
24
C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\fancyimpute\keras_models.py in <module>
15 from keras import backend as K
16 from keras import regularizers
---> 17 from keras.engine import Layer
18 from keras.initializers import RandomNormal
19
ImportError: cannot import name 'Layer'
Looks like progress at least.
Did you try running the following at the top of your notebook or in your IPython condole and restarting the kernel before running the rest of the code:
%pip install fancyimpute
I suspect you have automagics on in your IPython and so probably you already did the equivalent; howver, you are using a quite old Python there and so maybe the automagics isnāt on.
If that fails, re-try with the following:
%pip install --upgrade --force-reinstall fancyimpute
This suggests trying the following:
conda install -c conda-forge tensorflow
I have been racking my brain to understand how to properly do a fancyimpute installation and use this library for missing data analysis. If I were to uninstall Anaconda remove all the files and directories and start over. What would your suggestion be in order to do a error-free installation of fancyimpute library?
Okay, hereās what I have done so far.
- Uninstalled everything deleted all files related to Anaconda/Python.
- Install Microsoft C++ 2014+ Build Tools (as there was an error related to this).
- Reinstalled Anaconda 4.4.0 and Python 3.6 because fancyimpute seems to throw fewer errors with this configuration.
- Ran āpip install fancyimputeā
- It asked me to upgrade pip so I ran āpython -m pip install --upgrade pipā
- After this, I went to Jupyter Notebook, ran the following code:
import fancyimpute
#this time around hitting tab on Jupyter autofills for fancyimpute, it wasn't doing so before.
Hereās what I got:
ModuleNotFoundError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\keras\__init__.py in <module>()
2 try:
----> 3 from tensorflow.keras.layers.experimental.preprocessing import RandomRotation
4 except ImportError:
ModuleNotFoundError: No module named 'tensorflow.keras'
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
<ipython-input-1-9420e0b6af25> in <module>()
----> 1 import fancyimpute
C:\ProgramData\Anaconda3\lib\site-packages\fancyimpute\__init__.py in <module>()
3 from .solver import Solver
4 from .nuclear_norm_minimization import NuclearNormMinimization
----> 5 from .matrix_factorization import MatrixFactorization
6 from .iterative_svd import IterativeSVD
7 from .simple_fill import SimpleFill
C:\ProgramData\Anaconda3\lib\site-packages\fancyimpute\matrix_factorization.py in <module>()
12
13 import numpy as np
---> 14 from keras import regularizers
15 from keras.callbacks import EarlyStopping
16 from keras.layers import Input
C:\ProgramData\Anaconda3\lib\site-packages\keras\__init__.py in <module>()
4 except ImportError:
5 raise ImportError(
----> 6 'Keras requires TensorFlow 2.2 or higher. '
7 'Install TensorFlow via `pip install tensorflow`')
8
ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`
To rectify this, I ran
conda install -c conda-forge tensorflow
Hereās what I got:
>conda install -c conda-forge tensorflow
Fetching package metadata ...............
Solving package specifications:
InvalidSpecError: Invalid spec: =2.7
Any pointers for what I could do now?
Why didnāt you try installing a version close to what the error message says? Seems like you have a much newer version of Tensorflow (2.7 according to InvalidSpecError: Invalid spec: =2.7?) that isnāt compatible with the older versions of things you are using elsewhere. See here about specifying a specific version.
I would suggest this would be something to broach with the fancyimpute development and userās community. I donāt know if they have a discourse channel but they do have an issues page at Github.
It being tied to an older version of Python 3.6 (see below), means that issues will continue to mount.
Also, I note that what looks to be the fancyimpute github README, it clearly says:
Do not use conda. We donāt support it.
This reply also supports the view that pip is the way to go.
Maybe make a new conda environment where you just install pip via conda and try to install fancyimpute via pip?
For testing if that is the right way forward you could also do this using a repo at Github and a requirements.txt configuration like here. I would suggest not specifying versions first and see if import fancyimpute works. Also just start with with fancyimpute as the single line in requirements.txt and only add whatever it says it still needs. And definitely try it first by specifying in runtime.txt as Python 3.6. I just tried opening a Binder session from here and ran %pip install fancyimpute in a cell of a notebook, restarted the kernel. and then ran import fancyimpute in a cell of a notebook and saw it hang up in trying to get regularizers from Keras, too. Though the error was slightly different:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-1-3f82472c4160> in <module>
----> 1 import fancyimpute
/srv/conda/envs/notebook/lib/python3.8/site-packages/fancyimpute/__init__.py in <module>
3 from .solver import Solver
4 from .nuclear_norm_minimization import NuclearNormMinimization
----> 5 from .matrix_factorization import MatrixFactorization
6 from .iterative_svd import IterativeSVD
7 from .simple_fill import SimpleFill
/srv/conda/envs/notebook/lib/python3.8/site-packages/fancyimpute/matrix_factorization.py in <module>
12
13 import numpy as np
---> 14 from keras import regularizers
15 from keras.callbacks import EarlyStopping
16 from keras.layers import Input
/srv/conda/envs/notebook/lib/python3.8/site-packages/keras/__init__.py in <module>
18 from . import callbacks
19 from . import constraints
---> 20 from . import initializers
21 from . import metrics
22 from . import models
/srv/conda/envs/notebook/lib/python3.8/site-packages/keras/initializers/__init__.py in <module>
122 # from ALL_OBJECTS. We make no guarantees as to whether these objects will
123 # using their correct version.
--> 124 populate_deserializable_objects()
125 globals().update(LOCAL.ALL_OBJECTS)
126
/srv/conda/envs/notebook/lib/python3.8/site-packages/keras/initializers/__init__.py in populate_deserializable_objects()
80 v2_objs = {}
81 base_cls = initializers_v2.Initializer
---> 82 generic_utils.populate_dict_with_module_objects(
83 v2_objs,
84 [initializers_v2],
AttributeError: module 'keras.utils.generic_utils' has no attribute 'populate_dict_with_module_objects'
So maybe by using Python 3.6, you can avoid the issue that 3.8 has but testing that at this point would involve forking the repo and I leave it to you. Especially, since the fancyyimpute Github README states:
A variety of matrix completion and imputation algorithms implemented in Python 3.6.
I FINALLY FOUND A RATHER BARBARIC SOLUTION TO MY PROBLEM
But first @fomightez I would really like to thank you for patiently following my post and guiding me. The suggestions you gave led me to finally solve the problem with some minor adjustments. Here goes:
- Uninstalled Anaconda.
- Downloaded Visual Studio C++ Build Tools from here: https://aka.ms/vs/16/release/vc_redist.x64.exe
- Placed this dll called āmsvcp140_1ā in āC:\Users<username>\AppData\Roaming\Python\Python36ā folder as I got an error that the above mentioned file is missing from folders in the āPATHā (something along these lines).
- Deleted all the Anaconda related files from system folders.
- Reinstalled Anaconda version 4.4.0 and Python 3.6.
- Opened Anaconda terminal from inside the Anaconda Navigator.
- Ran the following commands:
#upgraded pip version
1. pythong -m pip install --upgrade pip
#installed tensorflow version 2.2 as the error asked for a version >=2,2
2. pip install tensorflow==2.2
#installed fancyimpute
3. pip install fancyimpute
If the above command throws below error:
ERROR: Cannot uninstall 'scikit-learn'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
then just delete the scikit-learn related files from Python36 and Anacond3 folders.
#in case you get the above-mentioned error, only then will this step applies to you.
4. pip install scikit-learn
This solved my problem.
1 Like
Thanks for posting the solution! Glad to know it worked out.
This is definitely a challenging one and atypical. (For you current notebook users with current Python, usually the %pip install or %conda install magic commands will help you avoid these older-style installation acrobatics. Or at least offer a word-around to get past them.)
Probably others will benefit from your thorough detailing of the solution.
