python - How can I improve my INSERT statement performance? -

-

How to insert a 256x64x250 value array in a MySQL database When I actually tried my INSERT statement on my data So it turned out very slow (in 6 minutes for a 16 minute file). Include "` data` (frame, sensor_ro, sensor_col, value) values ​​(% s,% s,% s,% s) for frames in range (NA): For rows in category (NA): For the Colonel Range (NX): cursor. Execute (query, (frame, rows, columns, data [rows, columns, frames]))

I was reading, who said that This was not the right approach because it is very difficult to execute 4 million separate inserts.

Now my data There is a lot of void (actually more than 90%), so I threw into the IF statement. Only enter more than zero value and I have used executemany () instead: For rows in the category (NA) in the `` `` `` `` `` `` `` `` `` `` `` `` `` `` `` `` For the rows in the range (nx) S,% s,% s,% s) "" "value = [] frame in range (nj): if data [rows, cols, frames] & gt; 0.0: value.append (frame, lines, columns, data [rows, columns, frames]) cur.executemany (query, value)

This miraculously brought down my processing time About 20 seconds, out of which in 14 seconds the list of values ​​ (37k rows) and 4 seconds is actually spent on formatting the database.

Now I am thinking, 'How can I move this process forward?' Because I think my loop is extremely inefficient and should be a better way. If I need to put 30 measurements for every dog, it can still take 10 minutes, which seems too long for the amount of this data.

Here are two versions of my raw files: or I would love to try Loading data INFILE, but I can not understand how to parse the data correctly.

If the data is a numpy array, you can try it: 

  query = "" include `data` (frame, sensor_ro, sensor_col, value) in values ​​(% s), rows, columns, rows, columns in frames, rows, columns, frames = numpy for frames. Nonzero (data) "" "" "" values ​​= [] rows:. Frame, line, call, data [row, call, frame]) cur.executemany (query, value)   
 or  
  query = "" Enter the values ​​in the data (frame, sensor_roo, sensor_col, value) (% s,% s,% s,% s) rows, cols, frame = numpy.nonzero (data) value = [(line, Columns, frames, data [rows, columns, frames])] for cur.executemany (query, value)    call, call, frame, call for wallet, frame, p > Hope this helps

Comments

Post a Comment

Popular posts from this blog

qt - switch/case statement in C++ with a QString type -

-
I want to use switch-case in my program but the compiler generates an error. How can I use the switch statement with QString ? The compiler gives me this error: The expression of the switch type 'QString' is invalid My code is as follows : boolStopWord (the word QString) {bool flag = false; Switch (word) {case "the": flag = true; break ; "On" case: flag = true; break ; Case "in": flag = true; break ; Case "your": flag = true; break ; Case "near": flag = true; break ; Case "all": flag = true; break ; Case "this": flag = true; break ; } Return flag; } How do I use the switch statement with a Caststring ? You can not use the switch statement in C ++ language with integral or enum types. You can formally enter the square type of object in the switch statement, but this means that the compiler will look for a user-defined conversion to convert it to an integral or enum type.
Read more

python - sqlite3.OperationalError: near "REFERENCES": syntax error - foreign key creating -

-
I'm trying to create a predefined key after checking the manual and I wrote it: Db.execute ("create table" Table 2 (id, integer primary key, somedata integer) "db.execute" (table creation table 2 does not exist (name text, maintain reference (id) ") and found this: sqlite 3. Operation error: near" references ": syntax error Your second ligne is wrong db.execute ("create table if table does not exist 2 (name text, my_id integer, foreign key (my_id) reference Maintenance (ID)) ") as explained
Read more

Python's equivalent for Ruby's define_method? -

-
What is a Python equivalent for Ruby's define_method , which is the dynamic generation of class methods will allow? (As can be seen in Wikipedia) Functions are first-class objects in Python and assign them The properties of a class or an example can be a way of doing the same thing as a Wikipedia example: colors = {"black": "000", "red": "F00", "green": "0f0", "yellow": "ff0", "blue": "00f", "magenta": "fff", "cyan": "0ff", "white": " FFF "} Class MyStream (ARR): Pass for Name, Code in colors.iteritems (): def _in_colour ( W code = code): return '& lt; Span style = "color:% s" & gt;% s & lt; / Span & gt; ' % (Code, self) setter (histrive, "in_" + name, _in_ color)
Read more