R performance with data reshaping -

-

I am trying to resize a data frame in R and use the suggested method to do this There is a problem in doing The data frame has the following structure: 

  id DATE1 DATE2 VALTYPE VALUE 'abcd1233' 2009-11-12 2009-12-23 'TYPE1' 123.45 ...

VALTYPE is a string and is a factor with only 2 values ​​( TYPE1 and TYPE2 ). I need to change the following data frame ("broad" relocation) based on common id and dates: 

  ID DATE1 DATE2 VALUE.TYPE1 VALUE.TYPE2 'abcd1233' 2009-11-12 2009-12-23 123.45 NA ...

There are more than 4,500,000 comments in the data frame (although about 70% VALUE s about NA < / Code>). The machine is Intel based Linux workstation with 4 GB RAM. By loading data (from a compressed Rdata file) into a fresh R process, it increases to about 250 MB, which obviously leaves a lot of space for the rescheduling.

These are my experiences so far:

, "DATE1", "DATE2"), TIMEWAYER = "VALTYPE");

RESULT: Error: size can not allocate vectors of 4.8 GB

  • Cast () Use the reshape package of the method:

    tbl2 & lt; - cast (tbl), id + DATE1 + DATE2 ~ VALTYPE);

    • RESULT: R process There is no end in the consumption of all RAM. Eventually this process had to be killed.

    • Using to () and merge () :

      SP & LT; - (TBL [C (1,2,3,5)], TBL $ walpie, function (x) x); TBL & LT; - Merge (SP [["TYPE1"]], SP [["TYPE2"]] = C ("ID", "DATE1", "DATE2"), all = true, sort = true);

    RESULT: Works fine, though it is not very elegant and stupid (i.e. it will break if more types are added).

    In order to add harm to the injury, in almost 3 lines of AOWK or Pearl (and, with hardly any RAM), the operation can be achieved, the question is whether all available What is a better way to do this operation using recommended methods without using RAM?

A useful move is to combine the id variables into a character vector and then to rearrange. 

  tbl $ NEWID & lt; - (tbl, paste (id, DATE1, DATE2, SP = ";" ")) TB2 and LT; - Restat (TB2, Nude ~ Valtipie, Measurement =" VALUE ")

In the same size problem it is almost 40% faster than the pair of my Intel Core 2 2.2ghz in the MacBook.


Comments

Post a Comment

Popular posts from this blog

c# - How to capture HTTP packet with SharpPcap -

-
I would like to capture all incoming HTTP packets on my machine. To do this, I am using SharpPcap which is a WinPcap The cover is. SharpPcap works very well, but it captures TCP packets and what it wants to do for me is a very low level. Does anyone know how can I easily get full HTTP requests / reactions from all these TCP packets? Thanks SharpPcap is already able to capture the packet in the same way That's what the virus does (just in the code instead of the GUI). And you can either either parse them directly or you can put them in the drive in the common .pcap file format. The steps to parse the capture are: Open a connection at any time Loop a while Or start capturing using an event callback Parsing the raw packet for the type you want If you are reading .PPP dump files In addition to calling you an offline capture reader is almost the same, there is no need to select an interface, and you can set the majority There is no need to do the modes. All st...
Read more

php - Multiple Select with Explode: only returns the word "Array" -

-
By using Wordpress I have created a multiple selection box so that users can select categories to exclude. When the page is initially loaded, I see my predefined default option. However, when I select a new value and I save ... I only see the word "Array" and have not chosen anything else? & lt; Class = "amultiple" id = "& lt; php echo $ value ['id'] ;? gt;" name = "& gt; php echo $ value ['id'] ;; gt; []" Multiple = "multiple" size = "8" & gt; & Lt ;? Php global option; Foreign currency ($ value as value $) {if (get_settings ($ value ['id']) === FALSE {$$ value ['id'] = $ value ['std']; } And {$$ value ['id'] = get_settings ($ value ['id']); }} $ Ranges = & amp; Amp; Categories ('type = post & orderbiz = name & amp; hide_empty = 1'); If ($ ranges) {$ ex_cat = implode (',', $ tt_cat_exclude); Forex Currency ($ range a...
Read more

php - jQuery AJAX Post not working -

-
I do not understand why this script does not work. It has something to do within AJAX PostScript / Function. Right now, when I submit my form, it runs php code on the same page. What it should do, send the values ​​of the form for Project_Azax.fp, then there will be a refund of success on the page which is correct or false. $ (document) .ready (function () {$ ('div # didIt'). Hide (); $ ('form [name = adminForm]'). Submit (function () ($ {'.post' ('/ project_ajax.php', {action: $ ('[name = action]') Val (), pId: $ ('[name = pId]'). Val (), name ('[Name = name]'. Val (), url: $ ('[url = url]'). Val (), summary: ('[summary = summary]'). Val ()}, Function (data) {if (data.success) {$ ('div # didIt') .Sliddown ('slow');} and {warning ('unsuccessful SA!');}}, 'Jason'); return false ;});}); What is the code for project_ajax.php below ... if ($ _ POST ['action'] == ...
Read more