My question is actually a quick one. I am using Spacy in a venv, following an online tutorial. I've just trained my model and whilst the training itself concluded, it also returns these errors:
(venv) [annemarie@annemarie-linux TrainingData]$ python3.8 -m spacy train config/config.cfg --paths.train corpus/spacy-docbins/train.spacy --paths.dev corpus/spacy-docbins/test.spacy --output output/models --training.eval_frequency 10 --training.max_steps 300
2022-07-28 08:35:25.488466: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-07-28 08:35:25.488481: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-07-28 08:35:26.409531: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-28 08:35:26.409662: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409694: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409721: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409747: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409773: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409799: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409824: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409850: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2022-07-28 08:35:26.409857: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
ℹ Saving to output directory: output/models
ℹ Using CPU
=========================== Initializing pipeline ===========================
[2022-07-28 08:35:26,788] [INFO] Set up nlp object from config
[2022-07-28 08:35:26,794] [INFO] Pipeline: ['ner']
[2022-07-28 08:35:26,797] [INFO] Created vocabulary
[2022-07-28 08:35:26,797] [INFO] Finished initializing nlp object
[2022-07-28 08:35:26,845] [INFO] Initialized pipeline components: ['ner']
✔ Initialized pipeline
============================= Training pipeline =============================
ℹ Pipeline: ['ner']
ℹ Initial learn rate: 0.001
E # LOSS NER ENTS_F ENTS_P ENTS_R SCORE
--- ------ -------- ------ ------ ------ ------
0 0 68.44 0.00 0.00 0.00 0.00
3 10 615.49 0.00 0.00 0.00 0.00
6 20 295.06 28.57 57.14 19.05 0.29
10 30 210.32 46.67 77.78 33.33 0.47
13 40 154.60 55.00 57.89 52.38 0.55
16 50 56.73 80.00 84.21 76.19 0.80
20 60 20.64 92.68 95.00 90.48 0.93
23 70 2.85 92.68 95.00 90.48 0.93
26 80 0.01 92.68 95.00 90.48 0.93
30 90 0.00 92.68 95.00 90.48 0.93
33 100 0.00 87.80 90.00 85.71 0.88
37 110 0.00 92.68 95.00 90.48 0.93
41 120 0.00 92.68 95.00 90.48 0.93
45 130 0.00 92.68 95.00 90.48 0.93
50 140 0.00 92.68 95.00 90.48 0.93
55 150 0.00 92.68 95.00 90.48 0.93
60 160 0.00 92.68 95.00 90.48 0.93
65 170 0.00 92.68 95.00 90.48 0.93
70 180 0.00 92.68 95.00 90.48 0.93
75 190 0.00 92.68 95.00 90.48 0.93
80 200 0.00 92.68 95.00 90.48 0.93
85 210 0.00 92.68 95.00 90.48 0.93
90 220 0.00 92.68 95.00 90.48 0.93
95 230 0.00 87.80 90.00 85.71 0.88
100 240 0.00 87.80 90.00 85.71 0.88
105 250 0.00 87.80 90.00 85.71 0.88
110 260 0.00 87.80 90.00 85.71 0.88
115 270 0.00 92.68 95.00 90.48 0.93
120 280 0.00 92.68 95.00 90.48 0.93
125 290 0.00 92.68 95.00 90.48 0.93
130 300 0.00 92.68 95.00 90.48 0.93
✔ Saved pipeline to output directory
output/models/model-last
Now, I understand I could probably just install 'libcudart.so.11.0'. However, neither the tutorial nor any others (or the doc) I looked into about Spacy mentioned I'd need to separately install this, so I am confused as to maybe I have done something wrong or installed something faulty.
I installed spacy using pip and I am using Python3.8, I'm on Manjaro (Arch). Should I just install libcudart? I've never worked much with python and this is my first time using spacy so I apologize if this question is weird.
EDIT: Which makes it more confusing, testing the model shows it functions just as I trained it. However, It'll give the same block of warnings as above for spacy.load("pathtomodel"), though.
CodePudding user response：
These warnings are related to tensorflow, which isn't being used spacy or spacy train for these pipelines.
spacy automatically imports tensorflow if it's available and that's why you see the warnings, but they shouldn't affect anything for typical spacy usage.
If you work in a venv where tensorflow is not installed, the warnings should go away.