Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentence compare alternate impl #1612

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Sentence compare alternate impl #1612

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1fdf4a5
implement word vector based search for future cluster creation.
capaximperii Feb 4, 2019
357dbc3
refactor searching to enable clustering in algo impl files.
capaximperii Feb 5, 2019
33c7fd3
proof of concept for Use wordvec clustering to speed up responses.
capaximperii Feb 7, 2019
56ecd1e
dont ignore sentences that contain only stopwords for a more natural …
capaximperii Feb 8, 2019
dd22544
refactor code to break less search framework.
capaximperii Feb 8, 2019
b65d53a
Save calculations to disk for optimised response speed.
capaximperii Feb 11, 2019
12bcea9
hack Sqlite bottleneck by saving statements to a text file before mod…
capaximperii Feb 13, 2019
f54a12a
Enabling some debug messages for benchmarking.
capaximperii Feb 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Enabling some debug messages for benchmarking.
  • Loading branch information
@capaximperii
capaximperii committed Feb 14, 2019
commit f54a12aad20f384428a0fb862104c860bcd17d41
10 changes: 5 additions & 5 deletions chatterbot/comparisons/embedded_wordvector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

logger = logging.getLogger(__name__)

NUM_CLUSTERS=10
NUM_CLUSTERS=20


class EmbeddedWordVector(Comparator):
Expand Down Expand Up @@ -169,7 +169,7 @@ def clusters_create_or_load(self):
for c in self.cluster_indices_group(i, self.model.labels_):
self.clusters[i].add(self.lines[c])
sz = len(self.clusters[i])
logger.debug(f"bucket {i} has {sz} elements")
logger.critical(f"bucket {i} has {sz} elements")
return True


Expand All @@ -196,7 +196,7 @@ def compare(self, statement, bot_statements_list):
index = self.model.predict(v)[0]
if index == 0: #prediction failed
index = randint(1, NUM_CLUSTERS)
logger.debug(f"predicted {index} for input")
logger.critical(f"predicted {index} for input")

min_match = 0.0
m_confidence = 100.0
Expand All @@ -212,10 +212,10 @@ def compare(self, statement, bot_statements_list):
if dist < m_confidence:
m_confidence = dist
m_statement = vals
logger.debug(f"compared {ii} values")
logger.critical(f"compared {ii} values")
m_statement.confidence = 0.0
if min_match != 0.0:
m_statement.confidence = 1.0 - m_confidence / min_match

logger.debug(f"Closest match found {m_statement.text}")
logger.critical(f"Closest match found {m_statement.text} @ {m_statement.confidence}")
return m_statement