[BUG] `No ModuleNotFoundError: No module named 'pkg_resources'`

[chillerb]

What issues are you experiencing?

Hi, I just discovered cytomulate and PyCytoData - very cool! Unfortunately, I got this error:

>>> import PyCytoData
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bjarne/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/PyCytoData/__init__.py", line 1, in <module>
    from PyCytoData.data import DataLoader, PyCytoData, FileIO
  File "/home/bjarne/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/PyCytoData/data.py", line 15, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

The issue seems to be caused by relying on an old version of setuptools, as pkg_resources is deprecated in setuptools>=82.

Expected behavior

Importing PyCytoData does not raise an error.

Your environment:

• OS: Debian
• Python Version: 3.12
• Python Distribution: UV
• PyCytoData version: '0.1.3'
• PyCytoData distribution: PyPI

[chillerb]

Also, it depends on numpy<2 :)

[kevin931]

Thank you for raising this issue! I've implemented a quick fix, which is currently tracked #27.

At the same time, can you elaborate on the numpy dependency? Is it from cytomulate?

[chillerb]

Hi @kevin931
sorry, I should have been more specific. I tried running this following the cytomulate tutorial notebook:

from PyCytoData import DataLoader

data = DataLoader.load_dataset(dataset="levine13")

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[10], line 3
      1 from PyCytoData import DataLoader
      2 
----> 3 data = DataLoader.load_dataset(dataset="levine13")

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/PyCytoData/data.py:871, in DataLoader.load_dataset(cls, dataset, sample, force_download, preprocess)
    868 if not os.path.exists(cls._data_path[dataset]):
    869     cls._download_data(dataset = dataset, force_download = force_download)   
--> 871 data: PyCytoData = cls._preprocess(dataset)
    873 if sample is not None and not isinstance(sample, np.ndarray):
    874     sample = np.array(sample).flatten()

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/PyCytoData/data.py:925, in DataLoader._preprocess(cls, dataset)
    923     fcs = cls._data_path[dataset] + "Levine_13dim_notransform.fcs"
    924     metadata = cls._data_path[dataset] + "Levine_13dim_cell_types.txt"
--> 925     data = cls._preprocess_levine13(fcs, metadata)
    926 elif dataset == "levine32":
    927     fcs = cls._data_path[dataset] + "Levine_32dim_notransform.fcs"

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/PyCytoData/data.py:944, in DataLoader._preprocess_levine13(cls, fcs, metadata)
    942 """Preprocess the Levine13 dataset."""
    943 df: pd.DataFrame
--> 944 _, df = fcsparser.parse(fcs, reformat_meta=True)
    945 labels: np.ndarray = np.loadtxt(metadata, delimiter="\t", dtype=str)
    947 labels = df["label"].apply(lambda x: "unassigned" if np.isnan(x) else labels[int(x),1]).to_numpy()

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/fcsparser/api.py:668, in parse(path, meta_data_only, compensate, channel_naming, reformat_meta, data_set, dtype, encoding)
    664     raise ParserFeatureNotImplementedError(u'Compensation has not been implemented yet.')
    666 read_data = not meta_data_only
--> 668 fcs_parser = FCSParser(path, read_data=read_data, channel_naming=channel_naming,
    669                        data_set=data_set, encoding=encoding)
    671 if reformat_meta:
    672     fcs_parser.reformat_meta()

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/fcsparser/api.py:185, in FCSParser.__init__(self, path, read_data, channel_naming, data_set, encoding)
    183 if path:
    184     with open(path, 'rb') as f:
--> 185         self.load_file(f, data_set=data_set, read_data=read_data)

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/fcsparser/api.py:210, in FCSParser.load_file(self, file_handle, data_set, read_data)
    208         break
    209 if read_data:
--> 210     self.read_data(file_handle)

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/fcsparser/api.py:521, in FCSParser.read_data(self, file_handle)
    518 native_code = '<' if (sys.byteorder == 'little') else '>'
    519 if endian != native_code:
    520     # swaps the actual bytes and also the endianness
--> 521     data = data.byteswap().newbyteorder()
    523 # Mask off high bits if integer type data
    524 if text["$DATATYPE"] == "I":

AttributeError: 'numpy.ndarray' object has no attribute 'newbyteorder'

Downgrading to numpy<2 resolved the issue.

However, I also noted another issue with cytomulate's Emulation mode:

from cytomulate import EmulationCytofData

cytof_data = EmulationCytofData()
cytof_data.initialize_cell_types(expression_matrix=data.expression_matrix, labels=data.cell_types)
expression_matrices, labels, _, _ = cytof_data.sample(n_samples=100)

---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_gaussian_mixture.py:364, in _compute_precision_cholesky(covariances, covariance_type, xp)
    363 try:
--> 364     cov_chol = _cholesky(covariance, xp)
    365 # catch only numpy exceptions, b/c exceptions aren't part of array api spec

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/utils/_array_api.py:1152, in _cholesky(covariance, xp)
   1151 if _is_numpy_namespace(xp):
-> 1152     return scipy.linalg.cholesky(covariance, lower=True)
   1153 else:

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/scipy/_lib/_util.py:1181, in _apply_over_batch.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
   1180 if not any(batch_shapes):
-> 1181     return f(*arrays, *other_args, **kwargs)
   1183 # Determine broadcasted batch shape

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/scipy/linalg/_decomp_cholesky.py:106, in cholesky(a, lower, overwrite_a, check_finite)
     52 """
     53 Compute the Cholesky decomposition of a matrix.
     54 
   (...)    104 
    105 """
--> 106 c, lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True,
    107                      check_finite=check_finite)
    108 return c

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/scipy/linalg/_decomp_cholesky.py:39, in _cholesky(a, lower, overwrite_a, clean, check_finite)
     38 if info > 0:
---> 39     raise LinAlgError(
     40         f"{info}-th leading minor of the array is not positive definite"
     41     )
     42 if info < 0:

LinAlgError: 9-th leading minor of the array is not positive definite

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[13], line 4
      1 from cytomulate import EmulationCytofData
      2 
      3 cytof_data = EmulationCytofData()
----> 4 cytof_data.initialize_cell_types(expression_matrix=data.expression_matrix, labels=data.cell_types)
      5 expression_matrices, labels, _, _ = cytof_data.sample(n_samples=100)

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/cytomulate/emulation/cytof_data.py:100, in EmulationCytofData.initialize_cell_types(self, expression_matrix, labels, max_components, min_components, covariance_types)
     97 ind = np.where(labels == c_type)[0]
     98 D = expression_matrix[ind, :]
--> 100 self.cell_types[c_type].fit(data=D,
    101                             max_components=max_components,
    102                             min_components=min_components,
    103                             covariance_types=covariance_types)
    105 cell_id += 1

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/cytomulate/emulation/cell_type.py:73, in EmulationCellType.fit(self, data, max_components, min_components, covariance_types)
     70 for n_components in range(min_components, max_components + 1):
     71     for cv_type in covariance_types:
     72         gm = GaussianMixture(n_components=n_components,
---> 73                              covariance_type=cv_type).fit(data)
     74         current_bic = gm.bic(data)
     75         if current_bic < smallest_bic:

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_base.py:199, in BaseMixture.fit(self, X, y)
    173 """Estimate model parameters with the EM algorithm.
    174 
    175 The method fits the model ``n_init`` times and sets the parameters with
   (...)    196     The fitted mixture.
    197 """
    198 # parameters are validated in fit_predict
--> 199 self.fit_predict(X, y)
    200 return self

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/base.py:1336, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1329     estimator._validate_params()
   1331 with config_context(
   1332     skip_parameter_validation=(
   1333         prefer_skip_nested_validation or global_skip_validation
   1334     )
   1335 ):
-> 1336     return fit_method(estimator, *args, **kwargs)

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_base.py:255, in BaseMixture.fit_predict(self, X, y)
    252 self._print_verbose_msg_init_beg(init)
    254 if do_init:
--> 255     self._initialize_parameters(X, random_state, xp=xp)
    257 lower_bound = -xp.inf if do_init else self.lower_bound_
    258 current_lower_bounds = []

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_gaussian_mixture.py:845, in GaussianMixture._initialize_parameters(self, X, random_state, xp)
    839 compute_resp = (
    840     self.weights_init is None
    841     or self.means_init is None
    842     or self.precisions_init is None
    843 )
    844 if compute_resp:
--> 845     super()._initialize_parameters(X, random_state, xp=xp)
    846 else:
    847     self._initialize(X, None, xp=xp)

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_base.py:158, in BaseMixture._initialize_parameters(self, X, random_state, xp)
    151     _, indices = kmeans_plusplus(
    152         X,
    153         self.n_components,
    154         random_state=random_state,
    155     )
    156     resp[indices, np.arange(self.n_components)] = 1
--> 158 self._initialize(X, resp)

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_gaussian_mixture.py:875, in GaussianMixture._initialize(self, X, resp, xp)
    873 if self.precisions_init is None:
    874     self.covariances_ = covariances
--> 875     self.precisions_cholesky_ = _compute_precision_cholesky(
    876         covariances, self.covariance_type, xp=xp
    877     )
    878 else:
    879     self.precisions_cholesky_ = _compute_precision_cholesky_from_precisions(
    880         self.precisions_init, self.covariance_type, xp=xp
    881     )

File ~/workspace/cytomulate-demo/.venv/lib/python3.12/site-packages/sklearn/mixture/_gaussian_mixture.py:367, in _compute_precision_cholesky(covariances, covariance_type, xp)
    365         # catch only numpy exceptions, b/c exceptions aren't part of array api spec
    366         except np.linalg.LinAlgError:
--> 367             raise ValueError(estimate_precision_error_message)
    368         precisions_chol[k, :, :] = _linalg_solve(
    369             cov_chol, xp.eye(n_features, dtype=dtype, device=device_), xp
    370         ).T
    371 elif covariance_type == "tied":

ValueError: Fitting the mixture model failed because some components have ill-defined empirical covariance (for instance caused by singleton or collapsed samples). Try to decrease the number of components, increase reg_covar, or scale the input data. The numerical accuracy can also be improved by passing float64 data instead of float32.

Casting the expression matrix to float64 actually resolved the issue:

from cytomulate import EmulationCytofData

cytof_data = EmulationCytofData()
cytof_data.initialize_cell_types(expression_matrix=data.expression_matrix.astype(np.float64), labels=data.cell_types)
expression_matrices, labels, _, _ = cytof_data.sample(n_samples=100)

[kevin931]

Ahh I see. For the former, it's an fcsparser issue, which is no longer being maintained. I will look for an alternative and document this in the interim. For the latter, your fix seems to work, and I can add to the documentation as well. Thanks for discovering these, and I will work on some improvements.