performance - What are the most efficient idioms for streaming data from disk with constant space usage? -

-

problem description

I need to stream large files from disk. I assume that file memory Being fit in is larger. Also, suppose I am doing some calculation on the data and the result is small enough to fit in memory. As a hypothetical example, suppose I need to compute an MD5 sequence of 200 GB files and guarantee me how much RAM will be used.

In summary:

    <
  • need constant space
  • as much as possible
  • Assume too large files
  • Result in memory

question

What are the fastest ways to read / read data from a file using a static location?

I had ideas

if the file was small enough to fit in memory, then mmap would be very fast on the POSIX system, unfortunately This is not the case here is there a performance gain for using mmap with a small buffer size to buffer the buffer of the file? Should the overhead be overwhelmed by any benefits to go down the system MMAP buffer, or should I use a fixed buffer that I have read with fread ?

I would not believe that mmap will be very fast (where very fast Defined with is much faster than fread .

Grep is used to use mmap , but fread one of the reasons was stability (strange things happen with MMAP, if the file is magged or an IO error occurs). There is some discussion about this history. >

you The grep can compare performance on your system with the - mmap option. The difference in performance on 200GB of file on my system is negligible, but your mileage may vary!

In short, I use fread with a fixed size buffer. It's easy to code, errors are easy to handle and almost certainly fast enough.


Comments

Post a Comment

Popular posts from this blog

c# - How to capture HTTP packet with SharpPcap -

-
I would like to capture all incoming HTTP packets on my machine. To do this, I am using SharpPcap which is a WinPcap The cover is. SharpPcap works very well, but it captures TCP packets and what it wants to do for me is a very low level. Does anyone know how can I easily get full HTTP requests / reactions from all these TCP packets? Thanks SharpPcap is already able to capture the packet in the same way That's what the virus does (just in the code instead of the GUI). And you can either either parse them directly or you can put them in the drive in the common .pcap file format. The steps to parse the capture are: Open a connection at any time Loop a while Or start capturing using an event callback Parsing the raw packet for the type you want If you are reading .PPP dump files In addition to calling you an offline capture reader is almost the same, there is no need to select an interface, and you can set the majority There is no need to do the modes. All st...
Read more

php - Multiple Select with Explode: only returns the word "Array" -

-
By using Wordpress I have created a multiple selection box so that users can select categories to exclude. When the page is initially loaded, I see my predefined default option. However, when I select a new value and I save ... I only see the word "Array" and have not chosen anything else? & lt; Class = "amultiple" id = "& lt; php echo $ value ['id'] ;? gt;" name = "& gt; php echo $ value ['id'] ;; gt; []" Multiple = "multiple" size = "8" & gt; & Lt ;? Php global option; Foreign currency ($ value as value $) {if (get_settings ($ value ['id']) === FALSE {$$ value ['id'] = $ value ['std']; } And {$$ value ['id'] = get_settings ($ value ['id']); }} $ Ranges = & amp; Amp; Categories ('type = post & orderbiz = name & amp; hide_empty = 1'); If ($ ranges) {$ ex_cat = implode (',', $ tt_cat_exclude); Forex Currency ($ range a...
Read more

php - jQuery AJAX Post not working -

-
I do not understand why this script does not work. It has something to do within AJAX PostScript / Function. Right now, when I submit my form, it runs php code on the same page. What it should do, send the values ​​of the form for Project_Azax.fp, then there will be a refund of success on the page which is correct or false. $ (document) .ready (function () {$ ('div # didIt'). Hide (); $ ('form [name = adminForm]'). Submit (function () ($ {'.post' ('/ project_ajax.php', {action: $ ('[name = action]') Val (), pId: $ ('[name = pId]'). Val (), name ('[Name = name]'. Val (), url: $ ('[url = url]'). Val (), summary: ('[summary = summary]'). Val ()}, Function (data) {if (data.success) {$ ('div # didIt') .Sliddown ('slow');} and {warning ('unsuccessful SA!');}}, 'Jason'); return false ;});}); What is the code for project_ajax.php below ... if ($ _ POST ['action'] == ...
Read more