1

i have a mysql database where i add news articles, and before adding to it it try to compere that article with 100 last articles if it has any similarity.

so if is 95% similar i can tag it as same as article 122 or if it is 70-95% similar i can tag it as similar to article 133,

Which is best way to do this:

  1. is there a way or a function that mysql can do it

  2. do i need to use python to compare that article in a while loop with other 100 articles

as i read in forums python is the best way, but i tried some library to compare string1(article1) with string2(article2) and even if its totally different article it tell me it is 70% same

i think it is because of some same words like : and , he ,she, will, news,text,or,and, the, i

artodm
  • 11
  • 1

2 Answers2

-1

as i read in forums python is the best way, but i tried some library to compare string1(article1) with string2(article2) and even if its totally different article it tell me it is 70% same

i think it is because of some same words like : and , he ,she, will, news,text,or,and, the, i

I will suggest to remove stopwords, might help.

SELECT * FROM INFORMATION_SCHEMA.INNODB_FT_DEFAULT_STOPWORD;

default mysql STOPWORDS. For Information look at the MYSQL Full-Text Stopwords Documentation & Fine-Tuning MySQL Full-Text Search.

mrbonewithgale
  • 100
  • 1
  • 7
-2

If you are using Linux you can call from python the diff command and play with the parameters, a teacher a few years ago did this to detect copy in a programing exam, it worked even after reformatting the code

RemedialGuns
  • 42
  • 8