c# - Compare the textual content of websites -

-

I am using a little experiment with text comparison / original literary theft, and website-to-website But I want to try it base However, I am searching in a proper way to process the text.

How will you process the content of two websites for plagiarism and compare them?

I'm thinking of something like this pseudo code: 

  // Websites to remove the text foreach website crawls website's stores, so pages are only all Scanned from the pages on the removal of text blocks - this is in the store list // compare in website1.textlist foreach text website 2. compare with all the text in the textlist

I know That this solution can collect a lot of data, so it's only It is with very small websites possible.

I have not yet decided on the actual text comparison algorithm, but I am still more interested in working the actual process algorithm first.

I 'I am thinking that it would be a good idea to remove all text in different text pieces (paragraphs, tables, headers and so on), as on text pages Can roam.

I am being mixed in C # (though ASP.NET).

I am very interested in any input or advice, so please shoot! :)

My approach to this problem will be for specific blocks of specific copyright, whose copyright Are trying to save.

If you want to prepare your own solution, here are some comments:

  • Respect robots.txt. If they have marked the site as non-crawl, then it is likely that they are not trying to profit from your content anyway.
  • You will need to refresh the site structure stored from time to time as websites change.
  • You must distinguish the text properly with HTML tags and JavaScript.
  • You will need to do a full text search in the entire text of the page tags / text for the extracted text) that are good, published algorithms for what you want to protect.

Comments

Post a Comment

Popular posts from this blog

c# - How to capture HTTP packet with SharpPcap -

-
I would like to capture all incoming HTTP packets on my machine. To do this, I am using SharpPcap which is a WinPcap The cover is. SharpPcap works very well, but it captures TCP packets and what it wants to do for me is a very low level. Does anyone know how can I easily get full HTTP requests / reactions from all these TCP packets? Thanks SharpPcap is already able to capture the packet in the same way That's what the virus does (just in the code instead of the GUI). And you can either either parse them directly or you can put them in the drive in the common .pcap file format. The steps to parse the capture are: Open a connection at any time Loop a while Or start capturing using an event callback Parsing the raw packet for the type you want If you are reading .PPP dump files In addition to calling you an offline capture reader is almost the same, there is no need to select an interface, and you can set the majority There is no need to do the modes. All st...
Read more

php - Multiple Select with Explode: only returns the word "Array" -

-
By using Wordpress I have created a multiple selection box so that users can select categories to exclude. When the page is initially loaded, I see my predefined default option. However, when I select a new value and I save ... I only see the word "Array" and have not chosen anything else? & lt; Class = "amultiple" id = "& lt; php echo $ value ['id'] ;? gt;" name = "& gt; php echo $ value ['id'] ;; gt; []" Multiple = "multiple" size = "8" & gt; & Lt ;? Php global option; Foreign currency ($ value as value $) {if (get_settings ($ value ['id']) === FALSE {$$ value ['id'] = $ value ['std']; } And {$$ value ['id'] = get_settings ($ value ['id']); }} $ Ranges = & amp; Amp; Categories ('type = post & orderbiz = name & amp; hide_empty = 1'); If ($ ranges) {$ ex_cat = implode (',', $ tt_cat_exclude); Forex Currency ($ range a...
Read more

jquery - SimpleModal Confirm fails to submit form -

-
I am trying to confirm jQuery to show yes / no confirmation when I submit a form. I had modified (in the last list) to capture and block form submissions. However, it only works with jQuery 1.2.6, a recent upgrade stops 1.3.2, if yes is clicked, then by submitting the form. But I can not understand what is wrong / inconsistent in my code now. Demo (which works in both versions of a simple window.href implementation). $ (document) .ready (function () {$ ('input.delete'). Click (function (e) {e.preventDefault (); var target = $ (e .target); Confirm ("Really delete this event?", Function () {target.click ()});});}); How about something like this? The idea of ​​catching clicks (and fire!) On the submitted event is not click event. $ (document) .ready (function () {$ ('# myForm'). Submit (function () {var f = this; Confirm (" Remove? ", Function () {$ (f). Submit ();}) Return; Return the initial session;}}}}; Hope That this is som...
Read more