Scala Concurrency slowing down -

-

I am doing to present with the fact that I am a relative java / ska newbie so i do not deny That something is clear which I am not doing.

I have got a Scala application that connects a MySQL database through Hibernate. This application is designed to process a large amount of data about 2,750,000 records, so I have tried to customize it as much as possible.

It's running on my workstation, which is a QuadCore Intel Xeon with 6 GB RAM (at 1033 MHz) and it runs well and quickly for the first 70 records; By the time it completes in 15 minutes, it has grown to 90k, it has been taken about 25 minutes, which is slowing down on some crawls.

I have checked the timer on the hibernate code and the database recovery is about the same as I have tried and tried to force manual garbage collection to do so, but it Not working too.

The code in question looks something like this: 

  val recordCount = Repo.recordCount batch size = 100 val batches = (to record 0 according to the batch size) .toList val for batchJobs = {(Batch Start & Lieutenant batches) yield (Future (new recordformer) format format (new repo (sessionFactory.openSession), batch start, batch size}} All (100000, batchjobs: * _ )

Inside the recordformer (which is not actually the name if you wondered on your naming plan madness ), It is another query to query for one of the next 100 records and then to pull back the actual record (between between start and end values), they are written in a text file as CSV Looking at the timer output, 5 seconds to pull back the record about each operation within the record operator and then output 0.1 file for one second.

This session is silent at regular intervals and is finally closing it. Run each recordAformer (each recordformer has its own session)

I'm mostly looking for any known bulk containing scala and futures. I have noticed that when it is slowing down, it does not use all eight potential threads which can definitely explain the decline in speed, but it is a mystery to me that why this will suddenly stop and always Around the record mark of 75

Thank you!

EDIT: It is an updated code to show that it uses yield and waits if all makes a difference.

jconsole application, which can be found in JDK ( $ JAVA_HOME / bin / Jconsole ) comes in the bundle with which the application runs it, it is very nice to tell you a few things:

  1. What apps do all your time by doing GC Used to be?
  2. What are the application threads doing?

Can you post the result here?


Comments

Post a Comment

Popular posts from this blog

c# - How to capture HTTP packet with SharpPcap -

-
I would like to capture all incoming HTTP packets on my machine. To do this, I am using SharpPcap which is a WinPcap The cover is. SharpPcap works very well, but it captures TCP packets and what it wants to do for me is a very low level. Does anyone know how can I easily get full HTTP requests / reactions from all these TCP packets? Thanks SharpPcap is already able to capture the packet in the same way That's what the virus does (just in the code instead of the GUI). And you can either either parse them directly or you can put them in the drive in the common .pcap file format. The steps to parse the capture are: Open a connection at any time Loop a while Or start capturing using an event callback Parsing the raw packet for the type you want If you are reading .PPP dump files In addition to calling you an offline capture reader is almost the same, there is no need to select an interface, and you can set the majority There is no need to do the modes. All st...
Read more

php - Multiple Select with Explode: only returns the word "Array" -

-
By using Wordpress I have created a multiple selection box so that users can select categories to exclude. When the page is initially loaded, I see my predefined default option. However, when I select a new value and I save ... I only see the word "Array" and have not chosen anything else? & lt; Class = "amultiple" id = "& lt; php echo $ value ['id'] ;? gt;" name = "& gt; php echo $ value ['id'] ;; gt; []" Multiple = "multiple" size = "8" & gt; & Lt ;? Php global option; Foreign currency ($ value as value $) {if (get_settings ($ value ['id']) === FALSE {$$ value ['id'] = $ value ['std']; } And {$$ value ['id'] = get_settings ($ value ['id']); }} $ Ranges = & amp; Amp; Categories ('type = post & orderbiz = name & amp; hide_empty = 1'); If ($ ranges) {$ ex_cat = implode (',', $ tt_cat_exclude); Forex Currency ($ range a...
Read more

php - jQuery AJAX Post not working -

-
I do not understand why this script does not work. It has something to do within AJAX PostScript / Function. Right now, when I submit my form, it runs php code on the same page. What it should do, send the values ​​of the form for Project_Azax.fp, then there will be a refund of success on the page which is correct or false. $ (document) .ready (function () {$ ('div # didIt'). Hide (); $ ('form [name = adminForm]'). Submit (function () ($ {'.post' ('/ project_ajax.php', {action: $ ('[name = action]') Val (), pId: $ ('[name = pId]'). Val (), name ('[Name = name]'. Val (), url: $ ('[url = url]'). Val (), summary: ('[summary = summary]'). Val ()}, Function (data) {if (data.success) {$ ('div # didIt') .Sliddown ('slow');} and {warning ('unsuccessful SA!');}}, 'Jason'); return false ;});}); What is the code for project_ajax.php below ... if ($ _ POST ['action'] == ...
Read more