Is there a way to broadcast ndarray of different shapes into a single ndarray?-CodePudding

Here a simple snippet of what I mean:

nested = np.array([[1,2,3],[4,5,6]])

scalar1= np.array([[1],[2]])
scalar2= np.array([[1],[3]])
scalar3= np.array([[1],[4]])

nesting = np.array([nested,scalar1,scalar2],dtype='object')

Here the error printed:

ValueError: could not broadcast input array from shape (2,3) into shape (2,)

About the context:

I'm working on a Fake News Detector of tweets, using a Neural Net that takes 2 inputs, a text sequence input, xtrain_pad of shape (1300,1500), and a numerical static input, X_train_static of shape (1300,7). Give this as two separate inputs to the Keras model I'm using it's not a problem, you can just put that as a list:

model.fit([xtrain_pad,X_train_static], y_train, 
                        epochs=epochs, 
                        batch_size=batch_size, 
                        validation_split=0.1, 
                        verbose=0,
                        validation_data=([xval_pad,X_val_static, y_val]),
                        callbacks=callback)

But now I'm trying to estimate the importance of the features using Shap. To do so the constructor to build a Shap explainer, like DeepExplainer is:

DE = shap.DeepExplainer(model, data = [xtrain_pad,X_train_static])

But the method accept as data only numpy.array or pandas.DataFrame format, and here my problem to transform 2 inputs into a single ndarray.

I've searched alternative approaches to tackle this, like using DataFrame or searching other Shap explainer, but none seems to work.

CodePudding user response：

What you are trying to do (I think) is concatenating or stacking not broadcasting.

If you had searched for how to concatenate or stack arrays horizontally you would probably have found:

nesting = np.hstack([nested,scalar1,scalar2])

nesting = np.concatenate([nested,scalar1,scalar2], axis=1)

Note: Bear in mind these operations will make a copy of the data though so make sure you have enough memory...

CodePudding user response：

In [40]: nested = np.array([[1,2,3],[4,5,6]])
    ...: scalar1= np.array([[1],[2]])
    ...: scalar2= np.array([[1],[3]])
    ...: scalar3= np.array([[1],[4]])

You have a (2,3) and (2,1) arrays.

When the component arrays have a matching first dimensions we get errors like this:

In [41]: np.array([nested, scalar1], object)
Traceback (most recent call last):
  Input In [41] in <cell line: 1>
    np.array([nested, scalar1], object)
ValueError: could not broadcast input array from shape (2,3) into shape (2,)

That broadcasting error is internal to np.array, a result of trying to map the first array, the (2,3) one, into a target of shape (2,). It's a bit obscure, but just says, using np.array this way does not work.

When combining (2,3),(2,1) and (1,2) it works. The arrays don't all match on the first dimension.

In [42]: np.array([nested, scalar1, scalar2.T], object)
Out[42]: 
array([array([[1, 2, 3],
              [4, 5, 6]]), array([[1],
                                  [2]]), array([[1, 3]])], dtype=object)

When the shapes match, the result can be 3d, not 1d object:

In [43]: np.array([scalar1, scalar2], object)
Out[43]: 
array([[[1],
        [2]],

       [[1],
        [3]]], dtype=object)
In [44]: _.shape
Out[44]: (2, 2, 1)

To reliably make an object dtype array of arrays you have to do something like:

In [45]: res = np.empty(4, object)
In [46]: res[:] = [nested, scalar1, scalar2, scalar3]
In [47]: res
Out[47]: 
array([array([[1, 2, 3],
              [4, 5, 6]]), array([[1],
                                  [2]]), array([[1],
                                                [3]]), array([[1],
                                                              [4]])],
      dtype=object)

However the fit might not like that. Usually it expects a n-d array with numeric dtype. You could for example, combine the 'scalar' into one (2,3) arrays, and then join that with the other (2,3) to make a (4,3), or a (2,2,3):

In [48]: np.hstack((scalar1, scalar2, scalar3))
Out[48]: 
array([[1, 1, 1],
       [2, 3, 4]])
In [49]: np.vstack((nested, _))
Out[49]: 
array([[1, 2, 3],
       [4, 5, 6],
       [1, 1, 1],
       [2, 3, 4]])

Or you might have to pad the 'scalar' to (2,3), and make a (4,2,3) from that.

Or joining them all along the horizontal axis

In [50]: np.hstack((nested,scalar1, scalar2, scalar3))
Out[50]: 
array([[1, 2, 3, 1, 1, 1],
       [4, 5, 6, 2, 3, 4]])