|
17 Apr 2006, 03:29
|
#1
|
Flash in the PAN
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
|
New Fast Dump Processing Code
I've put together a bundle of my recently completed dump processing php scripts.
The slowest - the planet processor, completes in well under 2 seconds on my entry level webhosting.
Note that these automatically calculate ALL ranks as well.
Included is an sql file for the database, and the php code.
Download it at http://www.fxmy.co.uk/pa/dump.zip
The code is a tiny bit messy, and its not commented. If you arent proficient with php just make the tables and run it, and the data is in the mysql ready to go
Updates:
* Fixed a minor bug with alliance ranks.
* All 3 Dumps are backed up to a 'dumps' folder each tick. (Thanks Banned for the idea). Be sure to CHMOD777 a dumps folder or this will not work.
Last edited by Shyne; 17 Apr 2006 at 13:24.
|
|
|
17 Apr 2006, 10:07
|
#2
|
Banned
Join Date: Jul 2003
Location: ******
Posts: 2,326
|
Re: New Fast Dump Processing Code
MySQL
I've been playing around with the same recently, but in pl/pgsql. Mine takes about 7-8 seconds to run on my p3-700, with about half the time being spent generating ranks for planets. I might try to improve this (I'm quite sure it's possible), but at the moment I have other stuff I'd prefer to work on.
Sharing tech is I mean to release my tech stuff at some point, but really, it requires a little more than a normal shell account. It runs python/postgresql, and stores every dump.
|
|
|
17 Apr 2006, 11:51
|
#3
|
Flash in the PAN
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
|
Re: New Fast Dump Processing Code
Well, the code should work fine on postgresql with little adaptation. If theres any demand ill convert it.
I know a lot of people even now are working on alliance tech applications and things, and seeing as I was working on this anyway I thought I'd share it.
If anyone has suggestions for improvements, let me know
|
|
|
18 Apr 2006, 22:49
|
#4
|
Wearer of The Hat
Join Date: Mar 2001
Location: Bedfordshire, UK
Posts: 252
|
Re: New Fast Dump Processing Code
Any chance you can add the code for inc/include.php?
from my basic knowledge of php in guessing it defines the various do-dahs (technical term there) like LISTING_LOCATION
edit: thanks, something like this has been missing for a while!
|
|
|
19 Apr 2006, 00:17
|
#5
|
Banned
Join Date: Jul 2003
Location: ******
Posts: 2,326
|
Re: New Fast Dump Processing Code
Quote:
Originally Posted by Shyne
* All 3 Dumps are backed up to a 'dumps' folder each tick. (Thanks Banned for the idea). Be sure to CHMOD777 a dumps folder or this will not work.
|
Code:
patools16=> select count(*) from planet_dump ;
count
---------
4476164
(1 row)
That's my r16 database
The way I figure it, if you're going to use a db, you should do something actually takes advantage of the db's primary purpose
This saving only one tick in the db bullocks is a bunch of hogwash.
|
|
|
19 Apr 2006, 00:28
|
#6
|
Flash in the PAN
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
|
Re: New Fast Dump Processing Code
:O
My current applications will only have interest in the latest tick. I do obviously see the use of a history table, but perhaps would need to investigate the best way of doing so.
A 4.4m row table might be rather slow - but having never used one, I'll leave it to you to comment
I heard that sandman says flatfile storage would have made his site faster than the current mysql site, but that may be untrue.
|
|
|
19 Apr 2006, 01:10
|
#7
|
Banned
Join Date: Jul 2003
Location: ******
Posts: 2,326
|
Re: New Fast Dump Processing Code
Quote:
Originally Posted by Shyne
My current applications will only have interest in the latest tick. I do obviously see the use of a history table, but perhaps would need to investigate the best way of doing so.
|
I don't actually use it for much in the way of history stuff. I do occasionally look up the old information. Plus I have a command for our IRC bot that shows score gained in the last 72 ticks. (So at worst the bot needs the last 72 ticks).
Quote:
A 4.4m row table might be rather slow - but having never used one, I'll leave it to you to comment
I heard that sandman says flatfile storage would have made his site faster than the current mysql site, but that may be untrue.
|
Flatfile will always be faster than a database application if you write the code right. With that said, the main bottleneck on our bot's speed is the IRC connection, which it throttles pretty heavily to avoid getting thrown off. I've occasionally run into problems when designing queries, but with the right indexes and structurings it's not a big deal.
|
|
|
19 Apr 2006, 12:31
|
#8
|
Emperor
Join Date: Jul 2001
Location: in front of a computer
Posts: 490
|
Re: New Fast Dump Processing Code
Quote:
Originally Posted by Banned
Flatfile will always be faster than a database application if you write the code right.
|
Jup, very much so. My own tools did use a packed binary data format stored in files and using a 512MB Ram Server meant it could have the whole 45000 ticks of a speedgame in memory or alternatively a whole round of PA with 188000 planets. As it allowed to create graphic charts of up to 9/10 different planets/galaxies/clusters showing roid/score growth in one dynamic chart on sometimes low-end hardware (300mhz servers), it needed to quickly access all data for some of those.
Quote:
Originally Posted by Banned
With that said, the main bottleneck on our bot's speed is the IRC connection, which it throttles pretty heavily to avoid getting thrown off. I've occasionally run into problems when designing queries, but with the right indexes and structurings it's not a big deal.
|
Thats why i gave up ordinary "query" connections to my old irc-bot interface. On networks like netgamers where it wasnt possible to run it as "service", i just told people to use the DCC chat interface instead. The flood restrictions on many irc networks are just too strict otherwise.
|
|
|
19 Apr 2006, 22:44
|
#9
|
Registered User
Join Date: Jul 2005
Location: Belgium
Posts: 30
|
Re: New Fast Dump Processing Code
For xVx I only keep the tick%24==0 ticks
I use those to calc score gain/loss etc.
I'd post our scripts here as well, but I need to clean them up a bit first
And they calc anything you'd like and put it in db (score, size, value, xp diffs, uni, para, cluster, gala ranks for planet, gala, cluster, para, rank diffs etc)
You get tables with a lot of columns that way
It does it all with MySQL queries though, using temp tables
TheShadow wrote some perl code once to do it in memory, but it didn't get finished
Maybe when I have some time I'll see if I can make all the calcs in PHP... It'll probably be faster then doing it with queries
|
|
|
20 Apr 2006, 00:01
|
#10
|
Flash in the PAN
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
|
Re: New Fast Dump Processing Code
Take a look at my code - personally I dont know a faster way of doing it (programatically at least). 2 seconds is adequate for me anyway, compared to my slower processor in the past.
Storing data at midnight is one way of doing it - statistics may tell you a best time of the day to take snapshots of planets
|
|
|
20 Apr 2006, 12:05
|
#11
|
Registered User
Join Date: Jul 2005
Location: Belgium
Posts: 30
|
Re: New Fast Dump Processing Code
2 seconds is a lot faster then wat our script takes atm, but calculating the universe ranks is the fastest/easiest anyway, and I guess the time for those is also about 2 secs...
It's the calculation of planet ranks per galaxy etc that takes longest
And that part is also harder to do then just sorting the array on score etc...
There's probably a way, but I've been too lazy to find another one after I found a way to do it in a MySQL/PHP script
|
|
|
20 Apr 2006, 12:24
|
#12
|
Registered User
Join Date: May 2004
Posts: 134
|
Re: New Fast Dump Processing Code
Good code Shyne
Our dump parser stores every single tick and takes on the average 1.58 seconds to do the related stuff (php/mysql).
__________________
Omen
|
|
|
29 Jan 2007, 09:57
|
#13
|
Cabeza Coder
Join Date: Oct 2000
Posts: 212
|
Re: New Fast Dump Processing Code
Nice code.
I like it
You are processing exiles, are you?
__________________
Lockhead
Developer, Solutions Architect, DevOps Engineer
lockhead.net
Quote:
Round 24 Conspiracy HC Comment at my planet
<Germania> 4.9.1
<Germania> hes our top hostile
|
|
|
|
29 Jan 2007, 12:30
|
#14
|
Flash in the PAN
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
|
Re: New Fast Dump Processing Code
I'm glad all that stuff is working for you guys. The code should be fairly clear so you can add whatever information you like in there.
The ranks are calculated in memory using the sort function - it took a long time to get that working but its fast now.
As for processing exiles - I'm not sure what you mean. I did make something that logged new planets, and their out-of-protection tick though - bit mean eh !
|
|
|
29 Jan 2007, 13:06
|
#15
|
CRASHING BEATS 'N FANTASY
Join Date: Mar 2001
Location: Cold Country.
Posts: 1,912
|
Re: New Fast Dump Processing Code
Quote:
Originally Posted by Shyne
I'm glad all that stuff is working for you guys. The code should be fairly clear so you can add whatever information you like in there.
The ranks are calculated in memory using the sort function - it took a long time to get that working but its fast now.
As for processing exiles - I'm not sure what you mean. I did make something that logged new planets, and their out-of-protection tick though - bit mean eh !
|
What he wonders is whether your importer keeps track of all coords a planet has been at or not (and mines corresponding planet data accordingly).
__________________
Ią! Ią! Munin F'tagn! - [*scendancy]
|
|
|
29 Jan 2007, 23:35
|
#16
|
Flash in the PAN
Join Date: Aug 2005
Location: Birmingham, Romania
Posts: 554
|
Re: New Fast Dump Processing Code
I dont - but you quite easily could.
The data is updated in that code, and the planets table always reflects the universe.
|
|
|
7 Feb 2009, 22:29
|
#17
|
Denial
Join Date: Dec 2008
Posts: 41
|
Re: New Fast Dump Processing Code
Sorry to dig up an old thread, but anyone know where i can get this file or something similar? Thnks
|
|
|
7 Feb 2009, 22:47
|
#18
|
CRASHING BEATS 'N FANTASY
Join Date: Mar 2001
Location: Cold Country.
Posts: 1,912
|
Re: New Fast Dump Processing Code
Quote:
Originally Posted by Rio
Sorry to dig up an old thread, but anyone know where i can get this file or something similar? Thnks
|
No problem digging this up. I don't know what Shyne did with that file, but if you are looking for a pa dump file importer, check out Munin:
http://github.com/andreaja/munin/
__________________
Ią! Ią! Munin F'tagn! - [*scendancy]
|
|
|
23 Apr 2011, 15:05
|
#19
|
Registered User
Join Date: Jan 2011
Posts: 8
|
Re: New Fast Dump Processing Code
Does anyone have the said dump-read-script by anychance?
|
|
|
|
All times are GMT +1. The time now is 12:17.
| |