Import Fancyimpute error

I am facing the following error on Jupyter Notebook when attempting to import fancy impute please help
EDIT:
Machine details: Core i5-4200u, 8GB DDR3 RAM, NVIDIA GT750M Graphics
Environment Details: Running the latest Anaconda version, using Jupyter Notebook from inside the Anaconda application.
I am not sure if I should have attached the entire code. However, prior to running the import I had run pip install multiple times to upgrade numpy, keras, ecos etc as per suggestions available on StackOverflow etc.

import fancyimpute
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-3f82472c4160> in <module>
----> 1 import fancyimpute

~\AppData\Roaming\Python\Python38\site-packages\fancyimpute\__init__.py in <module>
      2 
      3 from .solver import Solver
----> 4 from .nuclear_norm_minimization import NuclearNormMinimization
      5 from .matrix_factorization import MatrixFactorization
      6 from .iterative_svd import IterativeSVD

~\AppData\Roaming\Python\Python38\site-packages\fancyimpute\nuclear_norm_minimization.py in <module>
     11 # limitations under the License.
     12 
---> 13 import cvxpy
     14 
     15 from .solver import Solver

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\__init__.py in <module>
     16 
     17 __version__ = "1.1.13"
---> 18 from cvxpy.atoms import *
     19 from cvxpy.constraints import NonPos, Zero, SOC, PSD
     20 from cvxpy.expressions.expression import Expression

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\__init__.py in <module>
     18 from cvxpy.atoms.dist_ratio import dist_ratio
     19 from cvxpy.atoms.eye_minus_inv import eye_minus_inv, resolvent
---> 20 from cvxpy.atoms.geo_mean import geo_mean
     21 from cvxpy.atoms.gen_lambda_max import gen_lambda_max
     22 from cvxpy.atoms.gmatmul import gmatmul

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\geo_mean.py in <module>
     18 import numpy as np
     19 import scipy.sparse as sp
---> 20 from cvxpy.utilities.power_tools import (fracify, decompose, approx_error, lower_bound,
     21                                          over_bound, prettydict)
     22 from typing import List, Optional, Tuple

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\utilities\power_tools.py in <module>
     16 
     17 from fractions import Fraction
---> 18 from cvxpy.atoms.affine.reshape import reshape
     19 from cvxpy.atoms.affine.vstack import vstack
     20 from cvxpy.constraints.second_order import SOC

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\affine\reshape.py in <module>
     16 
     17 from cvxpy.expressions.expression import Expression
---> 18 from cvxpy.atoms.affine.hstack import hstack
     19 from cvxpy.atoms.affine.affine_atom import AffAtom
     20 from typing import Tuple

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\affine\hstack.py in <module>
     17 
     18 import cvxpy.lin_ops.lin_utils as lu
---> 19 from cvxpy.atoms.affine.affine_atom import AffAtom
     20 import numpy as np
     21 

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\atoms\affine\affine_atom.py in <module>
     20 import cvxpy.lin_ops.lin_utils as lu
     21 from cvxpy.atoms.atom import Atom
---> 22 from cvxpy.cvxcore.python import canonInterface
     23 from cvxpy.expressions.constants import Constant
     24 from cvxpy.utilities import performance_utils as perf

~\AppData\Roaming\Python\Python38\site-packages\cvxpy\cvxcore\python\__init__.py in <module>
      1 # TODO(akshayka): This is a hack; the swig-auto-generated cvxcore.py
      2 # tries to import cvxcore as `from . import _cvxcore`
----> 3 import _cvxcore

ImportError: numpy.core.multiarray failed to import

The error:

Kenobi, we now have a more elegant method of posting traceback errors for a more civilized age of asking for help.
Please don’t use screenshots. And please don’t post a fraction of the information as it makes it harder for those in the community to help you.
For posting the traceback as copied text learn about posting code blocks between triple back ticks, see the ‘Block Code’ section here, in particular look at how to do it below the text that says “Another (short) example, to demonstrate how to wrap your code blocks”. Discourse makes this easy; however, if you feel the traceback is long you can post it in a gist or a code snippet site and link to it in your post as well.

Also please review Getting good answers to your questions.

You don’t describe how you installed Jupyter notebook? How you installed cvxpy? Conda or pip? What type of machine you are on? Etc.?

Testable code examples would also be very nice to include in the future. That way we can test things are working in a fresh environment. They don’t have to be your code even, just related examples that involve imports of the same items.


That being said, my guess based on the limited information provided seems to be incompatibility with packages in your environment. Based on an open similar issue in the Issues listing for cvxpy and forcing an update, I’d suggest starting by installing cvxpy again by running the following at the following in a new cell right near the top of your notebook:

%pip install --upgrade --force-reinstall cvxpy

Following running that code, restart the kernel for the notebook, and then try running your code again.

If that simple approach fails, based on a very similar issue on StackOveflow here try to make more of the packages consistent. Maybe something like this in a new cell at the top:

%pip install --upgrade --force-reinstall numpy
%pip install --upgrade --force-reinstall scipy
%pip install --upgrade --force-reinstall cvxpy
%pip install --upgrade --force-reinstall fancyimpute

The fresh installs will hopefully make everything current and compatible in the notebook’s environment.

1 Like

I ran the following trouble shoots

conda update --all
conda create -n py36 python=3.6 anaconda
conda activate 
conda install ecos
conda install CVXcanon
conda install -c cvxgrp scs
conda install scs
pip install fancyimpute

after this I ran

import fancyimpute

Now I am getting the following error:

ImportError                               Traceback (most recent call last)
<ipython-input-1-3f82472c4160> in <module>
----> 1 import fancyimpute

C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\fancyimpute\__init__.py in <module>
      3 from .solver import Solver
      4 from .nuclear_norm_minimization import NuclearNormMinimization
----> 5 from .matrix_factorization import MatrixFactorization
      6 from .iterative_svd import IterativeSVD
      7 from .simple_fill import SimpleFill

C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\fancyimpute\matrix_factorization.py in <module>
     20 from .common import import_from
     21 from .scaler import Scaler
---> 22 from .keras_models import KerasMatrixFactorizer
     23 from .solver import Solver
     24 

C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\fancyimpute\keras_models.py in <module>
     15 from keras import backend as K
     16 from keras import regularizers
---> 17 from keras.engine import Layer
     18 from keras.initializers import RandomNormal
     19 

ImportError: cannot import name 'Layer'

Looks like progress at least.
Did you try running the following at the top of your notebook or in your IPython condole and restarting the kernel before running the rest of the code:

%pip install fancyimpute

I suspect you have automagics on in your IPython and so probably you already did the equivalent; howver, you are using a quite old Python there and so maybe the automagics isn’t on.
If that fails, re-try with the following:

%pip install --upgrade --force-reinstall fancyimpute

This suggests trying the following:

conda install -c conda-forge tensorflow

I have been racking my brain to understand how to properly do a fancyimpute installation and use this library for missing data analysis. If I were to uninstall Anaconda remove all the files and directories and start over. What would your suggestion be in order to do a error-free installation of fancyimpute library?

Okay, here’s what I have done so far.

  1. Uninstalled everything deleted all files related to Anaconda/Python.
  2. Install Microsoft C++ 2014+ Build Tools (as there was an error related to this).
  3. Reinstalled Anaconda 4.4.0 and Python 3.6 because fancyimpute seems to throw fewer errors with this configuration.
  4. Ran “pip install fancyimpute”
  5. It asked me to upgrade pip so I ran “python -m pip install --upgrade pip”
  6. After this, I went to Jupyter Notebook, ran the following code:
import fancyimpute 
#this time around hitting tab on Jupyter autofills for fancyimpute, it wasn't doing so before. 

Here’s what I got:

ModuleNotFoundError                       Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\keras\__init__.py in <module>()
      2 try:
----> 3     from tensorflow.keras.layers.experimental.preprocessing import RandomRotation
      4 except ImportError:

ModuleNotFoundError: No module named 'tensorflow.keras'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-9420e0b6af25> in <module>()
----> 1 import fancyimpute

C:\ProgramData\Anaconda3\lib\site-packages\fancyimpute\__init__.py in <module>()
      3 from .solver import Solver
      4 from .nuclear_norm_minimization import NuclearNormMinimization
----> 5 from .matrix_factorization import MatrixFactorization
      6 from .iterative_svd import IterativeSVD
      7 from .simple_fill import SimpleFill

C:\ProgramData\Anaconda3\lib\site-packages\fancyimpute\matrix_factorization.py in <module>()
     12 
     13 import numpy as np
---> 14 from keras import regularizers
     15 from keras.callbacks import EarlyStopping
     16 from keras.layers import Input

C:\ProgramData\Anaconda3\lib\site-packages\keras\__init__.py in <module>()
      4 except ImportError:
      5     raise ImportError(
----> 6         'Keras requires TensorFlow 2.2 or higher. '
      7         'Install TensorFlow via `pip install tensorflow`')
      8 

ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`

To rectify this, I ran

conda install -c conda-forge tensorflow

Here’s what I got:

>conda install -c conda-forge tensorflow
Fetching package metadata ...............
Solving package specifications:

InvalidSpecError: Invalid spec: =2.7

Any pointers for what I could do now?

Why didn’t you try installing a version close to what the error message says? Seems like you have a much newer version of Tensorflow (2.7 according to InvalidSpecError: Invalid spec: =2.7?) that isn’t compatible with the older versions of things you are using elsewhere. See here about specifying a specific version.

I would suggest this would be something to broach with the fancyimpute development and user’s community. I don’t know if they have a discourse channel but they do have an issues page at Github.

It being tied to an older version of Python 3.6 (see below), means that issues will continue to mount.


Also, I note that what looks to be the fancyimpute github README, it clearly says:

Do not use conda. We don’t support it.

This reply also supports the view that pip is the way to go.

Maybe make a new conda environment where you just install pip via conda and try to install fancyimpute via pip?
For testing if that is the right way forward you could also do this using a repo at Github and a requirements.txt configuration like here. I would suggest not specifying versions first and see if import fancyimpute works. Also just start with with fancyimpute as the single line in requirements.txt and only add whatever it says it still needs. And definitely try it first by specifying in runtime.txt as Python 3.6. I just tried opening a Binder session from here and ran %pip install fancyimpute in a cell of a notebook, restarted the kernel. and then ran import fancyimpute in a cell of a notebook and saw it hang up in trying to get regularizers from Keras, too. Though the error was slightly different:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-3f82472c4160> in <module>
----> 1 import fancyimpute

/srv/conda/envs/notebook/lib/python3.8/site-packages/fancyimpute/__init__.py in <module>
      3 from .solver import Solver
      4 from .nuclear_norm_minimization import NuclearNormMinimization
----> 5 from .matrix_factorization import MatrixFactorization
      6 from .iterative_svd import IterativeSVD
      7 from .simple_fill import SimpleFill

/srv/conda/envs/notebook/lib/python3.8/site-packages/fancyimpute/matrix_factorization.py in <module>
     12 
     13 import numpy as np
---> 14 from keras import regularizers
     15 from keras.callbacks import EarlyStopping
     16 from keras.layers import Input

/srv/conda/envs/notebook/lib/python3.8/site-packages/keras/__init__.py in <module>
     18 from . import callbacks
     19 from . import constraints
---> 20 from . import initializers
     21 from . import metrics
     22 from . import models

/srv/conda/envs/notebook/lib/python3.8/site-packages/keras/initializers/__init__.py in <module>
    122 # from ALL_OBJECTS. We make no guarantees as to whether these objects will
    123 # using their correct version.
--> 124 populate_deserializable_objects()
    125 globals().update(LOCAL.ALL_OBJECTS)
    126 

/srv/conda/envs/notebook/lib/python3.8/site-packages/keras/initializers/__init__.py in populate_deserializable_objects()
     80     v2_objs = {}
     81     base_cls = initializers_v2.Initializer
---> 82     generic_utils.populate_dict_with_module_objects(
     83         v2_objs,
     84         [initializers_v2],

AttributeError: module 'keras.utils.generic_utils' has no attribute 'populate_dict_with_module_objects'

So maybe by using Python 3.6, you can avoid the issue that 3.8 has but testing that at this point would involve forking the repo and I leave it to you. Especially, since the fancyyimpute Github README states:

A variety of matrix completion and imputation algorithms implemented in Python 3.6.

I FINALLY FOUND A RATHER BARBARIC SOLUTION TO MY PROBLEM
But first @fomightez I would really like to thank you for patiently following my post and guiding me. The suggestions you gave led me to finally solve the problem with some minor adjustments. Here goes:

  1. Uninstalled Anaconda.
  2. Downloaded Visual Studio C++ Build Tools from here: https://aka.ms/vs/16/release/vc_redist.x64.exe
  3. Placed this dll called “msvcp140_1” in “C:\Users<username>\AppData\Roaming\Python\Python36” folder as I got an error that the above mentioned file is missing from folders in the “PATH” (something along these lines).
  4. Deleted all the Anaconda related files from system folders.
  5. Reinstalled Anaconda version 4.4.0 and Python 3.6.
  6. Opened Anaconda terminal from inside the Anaconda Navigator.
  7. Ran the following commands:
#upgraded pip version
1. pythong -m pip install --upgrade pip
#installed tensorflow version 2.2 as the error asked for a version >=2,2 
2. pip install tensorflow==2.2
#installed fancyimpute
3. pip install fancyimpute

If the above command throws below error:

ERROR: Cannot uninstall 'scikit-learn'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

then just delete the scikit-learn related files from Python36 and Anacond3 folders.

#in case you get the above-mentioned error, only then will this step applies to you. 
4. pip install scikit-learn

This solved my problem.

1 Like

Thanks for posting the solution! Glad to know it worked out.
This is definitely a challenging one and atypical. (For you current notebook users with current Python, usually the %pip install or %conda install magic commands will help you avoid these older-style installation acrobatics. Or at least offer a word-around to get past them.)
Probably others will benefit from your thorough detailing of the solution.