c# - Compare the textual content of websites -

-

I am using a little experiment with text comparison / original literary theft, and website-to-website But I want to try it base However, I am searching in a proper way to process the text.

How will you process the content of two websites for plagiarism and compare them?

I'm thinking of something like this pseudo code: 

  // Websites to remove the text foreach website crawls website's stores, so pages are only all Scanned from the pages on the removal of text blocks - this is in the store list // compare in website1.textlist foreach text website 2. compare with all the text in the textlist

I know That this solution can collect a lot of data, so it's only It is with very small websites possible.

I have not yet decided on the actual text comparison algorithm, but I am still more interested in working the actual process algorithm first.

I 'I am thinking that it would be a good idea to remove all text in different text pieces (paragraphs, tables, headers and so on), as on text pages Can roam.

I am being mixed in C # (though ASP.NET).

I am very interested in any input or advice, so please shoot! :)

My approach to this problem will be for specific blocks of specific copyright, whose copyright Are trying to save.

If you want to prepare your own solution, here are some comments:

  • Respect robots.txt. If they have marked the site as non-crawl, then it is likely that they are not trying to profit from your content anyway.
  • You will need to refresh the site structure stored from time to time as websites change.
  • You must distinguish the text properly with HTML tags and JavaScript.
  • You will need to do a full text search in the entire text of the page tags / text for the extracted text) that are good, published algorithms for what you want to protect.

Comments

Post a Comment

Popular posts from this blog

Is there an open source WebSockets (JavaScript) XMPP library? -

-
Has anyone written the open source XMPP library that uses the websacet and is being run by a browser? We do not yet have a standard XMPP that can enforce the server, Must be adequately faced before it may be necessary The first step is to eliminate WebSAT standardization. It seems that it can be in an IETF Haib working group, which has not been approved by the IESG so far in the writing time (see) a few weeks ago (Hiroshima), a hybrid bird-of-a-feather in IETF ( BOF), which went very well. WebSketches have a constant reference, and it seems to be systematically organized, one will make XMPP binding for webmasters, possibly with a bar for every webset frame. Edit: Jack Moffitt has written an IETF can be used with the first pass on a protocol. Warning . It is still likely to change fast enough, only apply it if you are ready to wrap it completely later. Warning
Read more

java - Is there an object like a "Set" that can contain only unique string values, but also contain a count on the number of occurrences of the string value? -

-
There is an object like "set" in Java that can contain only unique string values, but it also includes a count Is the number of string value in number? This idea is simple. A data set, ALA ... ABBCCC I will set the text of each line set- I want to add to the object every time a non-unique text is added to the set, I also want to show a numeric value associated with the set to show how often it was added. So if I run it on the above data then set output will be something like this: A: 1B: 2C: 3 Any ideas? Maps & lt; String, integer & gt; add public null (strings) {if (map.containsKey (s)) {map.put (s, map.get (s) + 1); } And {map.put (S, 1); }}
Read more

mysql - Can mysql_pconnect be called multiple times in one php page? -

-
I was wondering if it is valid for calling mysql_pconnect several times in a php page? Also, if possible, should I either close my last ISSL handle or something? Actually the following code is okay? Or is there a lack of some other steps? mysql_pconnect ("ip: 3306", "user", "pass") or die (mysql_error ()); Mysql_select_db ("test") or die (mysql_error ()); // Some SQL queries mysql_pconnect ("ip2: 3306", "user", "pass") or die (mysql_error ()); Mysql_select_db ("test") or die (mysql_error ()); // Another SQL query Yes but this is probably a better practice, both the headset To store both in the database and make the correct DB explicit use. dbServer1 = mysql_pconnect ("ip: 3306", "user", "pass") or die (mysql_error ()); Mysql_select_db ("test", dbServer1) or die (mysql_error ()); DbServer2 = mysql_pconnect ("ip: 3306", "user", "...
Read more