perl - mailserver log filtering -

-

I have a list of multi-GB messenger log files and ~ 350k message IDs. I want to pull out large log file lines with IDs from a long list ... and I want to sharpen it now ... I currently do it in Perl: 

  #! Use / usr / bin / perl warnings; Opening file with #list - Unique ID open id more than 350k; MyLog_ID; @lista_id = & lt; ID & gt; Closed id; Chomp @lista_id; Open log, malog; # While - External currency is out of memory (& lt; LOG & gt;) {$ wiersz = $ _; My @virus_split = partition ('', $ wiersz); # Foreach (@lista_id) {$ id = $ _; If #Id is the 6th column in Merral ($ wiersz_split [5] eq $ id) {# Print on Completion Match - STDOUT or File or Anything Could Print "@VirusSpplit \ n"; }}} Closed log;   
 It works but it is slow ... compared with the list of each line ID from the log. Should I use databases and join in a way? Or compare substrings?  
 There are a number of tools for log analysis - e.g. Pflogsumm ... but it summarizes just as it will be fast but useless and I'll use it after filtering my log file  
  grep -c "status = sent" maillog   
 .. this is for pflogsumm etc. - just the variable is moving.  
 Any suggestions?  
 ------------------- - UPDATE -------------------  
 Thank you Dallahlen, I have successfully ({code> @lista_id ) with it:  
  if ($ lista_id_hash exists {$ wiersz_split [5]}) { Print "$ wiersz"; }   
 Where % lista_id_hash  is a hash table where the keys are items taken from my id list. It works superfast to filter interesting logs to 4x6 GB log file & gt; Processing with 350k id takes less than 1 minute.   
 
 
 Use a hash  
  My% is known; Known for @lista_id {$ _} = 1; # ... while (& lt; & gt;) {# ... Determine ID ($ known {$ id}) {# processing line}; };   
 PS If your log is large, then you might be better off with sharing such as the last two letters of $ 25 (or 36 ** 2?) Small files a poor man Like a map of, at a time, the number of IDs stored in memory will also be reduced (i.e., when you are processing maillog.split.cf, you should just end the ID in "CF" in the hash).

Comments

Post a Comment

Popular posts from this blog

qt - switch/case statement in C++ with a QString type -

-
I want to use switch-case in my program but the compiler generates an error. How can I use the switch statement with QString ? The compiler gives me this error: The expression of the switch type 'QString' is invalid My code is as follows : boolStopWord (the word QString) {bool flag = false; Switch (word) {case "the": flag = true; break ; "On" case: flag = true; break ; Case "in": flag = true; break ; Case "your": flag = true; break ; Case "near": flag = true; break ; Case "all": flag = true; break ; Case "this": flag = true; break ; } Return flag; } How do I use the switch statement with a Caststring ? You can not use the switch statement in C ++ language with integral or enum types. You can formally enter the square type of object in the switch statement, but this means that the compiler will look for a user-defined conversion to convert it to an integral or enum type.
Read more

python - sqlite3.OperationalError: near "REFERENCES": syntax error - foreign key creating -

-
I'm trying to create a predefined key after checking the manual and I wrote it: Db.execute ("create table" Table 2 (id, integer primary key, somedata integer) "db.execute" (table creation table 2 does not exist (name text, maintain reference (id) ") and found this: sqlite 3. Operation error: near" references ": syntax error Your second ligne is wrong db.execute ("create table if table does not exist 2 (name text, my_id integer, foreign key (my_id) reference Maintenance (ID)) ") as explained
Read more

Python's equivalent for Ruby's define_method? -

-
What is a Python equivalent for Ruby's define_method , which is the dynamic generation of class methods will allow? (As can be seen in Wikipedia) Functions are first-class objects in Python and assign them The properties of a class or an example can be a way of doing the same thing as a Wikipedia example: colors = {"black": "000", "red": "F00", "green": "0f0", "yellow": "ff0", "blue": "00f", "magenta": "fff", "cyan": "0ff", "white": " FFF "} Class MyStream (ARR): Pass for Name, Code in colors.iteritems (): def _in_colour ( W code = code): return '& lt; Span style = "color:% s" & gt;% s & lt; / Span & gt; ' % (Code, self) setter (histrive, "in_" + name, _in_ color)
Read more