sklearn.feature_extraction.text.CountVectorizer
vocabulary
引数を介してボキャブラリーを渡すことでオブジェクトを取得しましたが、sklearn.utils.validation.NotFittedError: CountVectorizer - Vocabulary wasn't fitted.
エラーメッセージ。どうして?
例:
import sklearn.feature_extraction
import numpy as np
import pickle
# Save the vocabulary
ngram_size = 1
dictionary_filepath = 'my_unigram_dictionary'
vectorizer = sklearn.feature_extraction.text.CountVectorizer(ngram_range=(ngram_size,ngram_size), min_df=1)
corpus = ['This is the first document.',
'This is the second second document.',
'And the third one.',
'Is this the first document? This is right.',]
vect = vectorizer.fit(corpus)
print('vect.get_feature_names(): {0}'.format(vect.get_feature_names()))
pickle.dump(vect.vocabulary_, open(dictionary_filepath, 'w'))
# Load the vocabulary
vocabulary_to_load = pickle.load(open(dictionary_filepath, 'r'))
loaded_vectorizer = sklearn.feature_extraction.text.CountVectorizer(ngram_range=(ngram_size,ngram_size), min_df=1, vocabulary=vocabulary_to_load)
print('loaded_vectorizer.get_feature_names(): {0}'.format(loaded_vectorizer.get_feature_names()))
出力:
vect.get_feature_names(): [u'and', u'document', u'first', u'is', u'one', u'right', u'second', u'the', u'third', u'this']
Traceback (most recent call last):
File "C:\Users\Francky\Documents\GitHub\Adobe\dstc4\test\CountVectorizerSaveDic.py", line 22, in <module>
print('loaded_vectorizer.get_feature_names(): {0}'.format(loaded_vectorizer.get_feature_names()))
File "C:\Anaconda\lib\site-packages\sklearn\feature_extraction\text.py", line 890, in get_feature_names
self._check_vocabulary()
File "C:\Anaconda\lib\site-packages\sklearn\feature_extraction\text.py", line 271, in _check_vocabulary
check_is_fitted(self, 'vocabulary_', msg=msg),
File "C:\Anaconda\lib\site-packages\sklearn\utils\validation.py", line 627, in check_is_fitted
raise NotFittedError(msg % {'name': type(estimator).__name__})
sklearn.utils.validation.NotFittedError: CountVectorizer - Vocabulary wasn't fitted.
何らかの理由で、sklearn.feature_extraction.text.CountVectorizer()
の引数として_vocabulary=vocabulary_to_load
_を渡したとしても、loaded_vectorizer._validate_vocabulary()
を呼び出す前にloaded_vectorizer.get_feature_names()
を呼び出す必要があります。
したがって、例では、語彙を使用してCountVectorizerオブジェクトを作成するときに、次のことを行う必要があります。
_vocabulary_to_load = pickle.load(open(dictionary_filepath, 'r'))
loaded_vectorizer = sklearn.feature_extraction.text.CountVectorizer(ngram_range=(ngram_size,
ngram_size), min_df=1, vocabulary=vocabulary_to_load)
loaded_vectorizer._validate_vocabulary()
print('loaded_vectorizer.get_feature_names(): {0}'.
format(loaded_vectorizer.get_feature_names()))
_