Tools site revisited.

[Gayle29uk]

Ok, so I was thinking about adapting my tools site to DoM and having a browse through my PHP Developers Cookbook to check out how to do XML parsing and it came to me in a blinding flash!

Why adapt the site? Why not adapt the parser?

Most PA clones have a similar interface and pretty much the same info is needed for a tools site (coords, name, size, score etc) so why reinvent the wheel?

Why not just write a pre-parser for each supported game to create an XML file in a standard format from the DB dump and have that as the input? That way all that needs to be done to support a new game is to write a new pre-parser.

Yes? No? Shut up Gayle?

[wu_trax]

i think thats exactly what xml was invented for.
(but parsing html can be really horrible :/)

[Gayle29uk]

Originally posted by wu_trax
i think thats exactly what xml was invented for.
(but parsing html can be really horrible :/)

Parsing HTML? Or do you mean XML? In PHP4 it's ridiculously easy

[wu_trax]

ok, maybe i missunderstood something, how are you going to get the xml file?

[Gayle29uk]

Originally posted by wu_trax
ok, maybe i missunderstood something, how are you going to get the xml file?

I take the DB dump in whatever format it's in and parse it into an XML file.

[wu_trax]

ah ok, i thought the db-dumb is in html-format

(maybe it was a mistake never to play this game )

[queball]

Sounds ok. I don't see the advantage myself.

[W]

My question would be, what's the difference between a parser and a pre-parser? Where is there work saved?

[Add]

Originally posted by W
My question would be, what's the difference between a parser and a pre-parser? Where is there work saved?

i assume, the tools site would read a standard format of xml. When Gayle wants to add a new game, i again assume, she would write a 'pre-'parser to parse it's dump into her standard format, then simply 'load' the xml file into the tools site config and all of the information would be updated for the new game.

[Gayle29uk]

Originally posted by Add
i assume, the tools site would read a standard format of xml. When Gayle wants to add a new game, i again assume, she would write a 'pre-'parser to parse it's dump into her standard format, then simply 'load' the xml file into the tools site config and all of the information would be updated for the new game.

Exactly

And of course it's a perfect excuse to work with XML which I've never done before

[Add]

Originally posted by Gayle29uk
Exactly

And of course it's a perfect excuse to work with XML which I've never done before

it's practically useless. it's just a new toy for the úber geeks to play with.

[pablissimo]

Useless; bollocks.

It gives a nice standardised way of holding data in a context that doesn't rely on anything beyond comprehension of the XML format.

[Add]

Originally posted by pablissimo
Useless; bollocks.

It gives a nice standardised way of holding data in a context that doesn't rely on anything beyond comprehension of the XML format.

USELESS.

if i wanted to hold information in a format for humans to understand i would use a table.

[queball]

Originally posted by Add
i assume, the tools site would read a standard format of xml. When Gayle wants to add a new game, i again assume, she would write a 'pre-'parser to parse it's dump into her standard format, then simply 'load' the xml file into the tools site config and all of the information would be updated for the new game.

What's the actual advantage of using an external format (XML) instead of an internal one (an API)? Are you saying it's an advantage that the dump translators and the site code are seperate processes/codebases/whatever?

[pablissimo]

Originally posted by Add
USELESS.

if i wanted to hold information in a format for humans to understand i would use a table.

It's not meant for human digestion though.

It's meant to be easily parsable and structured in a way that can represent tabular data.

And it does this well.

[pablissimo]

Originally posted by queball
What's the actual advantage of using an external format (XML) instead of an internal one (an API)? Are you saying it's an advantage that the dump translators and the site code are seperate processes/codebases/whatever?

There's in-built functions for parsing of XML.

There aren't in-built functions for parsing of GayleML.

[queball]

Originally posted by pablissimo
It's not meant for human digestion though.

Yes it is, it's meant for both humans and machines, unless the w3c has changed its story recently (I wouldn't be surprised). People seem to forget this.

There aren't in-built functions for parsing of GayleML.

What are you talking about? An API deals with language constructs: integers, strings, associative arrays, etc. This generally doesn't need to be "parsed" (though galaxy names might be parsed for tags or whatever).

[pablissimo]

Originally posted by queball
Yes it is, it's meant for both humans and machines, unless the w3c has changed its story recently (I wouldn't be surprised). People seem to forget this.


What are you talking about? An API deals with language constructs: integers, strings, associative arrays, etc. This generally doesn't need to be "parsed" (though galaxy names might be parsed for tags or whatever).

If you're using a complex construction of data types, then it requires your data to be stored exactly. It also requires that you know the exact layout of your data structures in order to use them at a later date, or to have some form of adaption for other inputs, in this case 'some other PA clone'.

If you're using XML, which is a standard, you don't need to know anything other than how to write XML. This makes the entire system easier to release as source for someone else, it makes it more easily extensible by someone else in the short-term or even you in the longer-term when you've forgotten your complex structures.

It's simply another layer of abstraction.

[queball]

What are we really talking about though?

API version, each dump perhaps returns an object (with various arbitrary methods itself), with a function getrow(), returning something like array("planetname" => "XML", "rulername" => "hype", "score" => 10000). This is called until it returns null.

XML version, each dumpfile handler reads in its dumpfile and writes out
<planets>
<planet>
<planetname>file conversion</planetname>
<rulername>the triviality</rulername>
<score>20000</score>
</planet>
...
</planets>
Some other code reads this in, probably reading each planet into an associative array, and writes out some SQL.

Originally posted by pablissimo

If you're using a complex construction of data types, then it requires your data to be stored exactly. It also requires that you know the exact layout of your data structures in order to use them at a later date, or to have some form of adaption for other inputs, in this case 'some other PA clone'.

If you're using XML, which is a standard, you don't need to know anything other than how to write XML. This makes the entire system easier to release as source for someone else, it makes it more easily extensible by someone else in the short-term or even you in the longer-term when you've forgotten your complex structures.

It's simply another layer of abstraction.

OK, lies about XML #1: it is self-describing.
If the person reading it already knows what it's doing the tags can help, but since it is just a complicated version of a tree, it has no real advantage over a well labelled associative array for example.

Compare the two examples I gave; how can you justify claims of "standard" and "extensible"? How are arrays any less "standard" than XML? OK, if pre-parsers are going to be written in a language that doesn't integrate with PHP or can only write a stream of data then obviously you need some external format. So you might add an XML interface, since it's "standard" (commonly implemented). But that can come on top - I'm an XP kind of guy. If you're planning on doing something with long term utility, without language barriers, you should be using .NET.

Don't compare one file with lots of nice descriptive XML tags with some incomprehensible half-arsed PHP structure. There is nothing magical in XML that makes it easy to understand (seen how graphs get represented?). If you find it really easy to design great file formats as XML appilcations but struggle writing a usable data structure in your language of choice then fine, use XML. I'm asking for an advantage of writing all that extra code, to a competant coder who has written useful tools in the past, and is now deciding to spend her time converting one syntax to another.

[Gayle29uk]

Okay...let me explain the idea here...

Different games have different DB dump formats therefore adapting tools site to different games requires modification. Now imagine this as the process...

1. Download DB dump from wherever.
2. Pre-parse DB dump into an XML file containing x,y,group name,z,planet/tribe/whatever name,ruler/leader/whatever name,size,score. This is a fixed format and will be the same for all games.
3. A standard script picks this XML file up and does it's thing with the database/ranks/score changes etc. It doesn't care what the game is because the data is pre-parsed into a format it can understand.
4. Tools site shows data

Note that only steps 1 and 2 change for a new game (step 1 being a single or at most 2 lines of a bash script anyway), steps 3 and 4 will not change by even one line of code.

Add a 'we're playing THIS game' line to the config file (final distribution version) and you now have a working tools site that any idiot can set up for any supported game. New games are easy to add on because only one file has to be written to support a new one.

Or am I making life hard for myself?

[Raging.Retard]

Apart from increasing your knowledge of XML, I dont see how this alters any aspect of your exisiting system, apart from increased work.

I dont see how writing a data parser ouptutting to XML, which then writes to the databse will gain anything over parsing the data directly and then writing to the DB. In either case your sitll having to write the parser to extract the same data? Anything like planet ranking surely is not affect, as you would do that after the data had been stored? In which case its already in a standard format, that of your database.

[Nodrog]

Using XML for this data seems likely to be pretty horrible: wont your markup end up accounting for more than half the document size?

<stone>1334</stone>
<timber>5000</timber>


ugh

[mbushell]

Parsing xml is a piece of piss, nod: it's the markup what names the variables (in my case anyway), although csv would parse quicker.

Edit: Wouldn't it be better as <planet stone="2002" timber="1338"/> anyway?

[Nodrog]

Originally posted by mbushell
Parsing xml is a piece of piss, nod: it's the markup what names the variables (in my case anyway), although csv would parse quicker.

Edit: Wouldn't it be better as <planet stone="2002" timber="1338"/> anyway?

Using XML when it offers no tangible benefits and is going to result in markup being over 70% of the total document size is nothing more than mastabatory abuse of new technology, and is the reason why a lot of people hate XML in the first place.

[pablissimo]

True, there is insufficient benefit to counter the huge cost of markup (though that should compress well given there's lots of similar symbol strings).

The benefits that have been touted to me are that you can use a standardise set of input and output routines for any program/system you write because you always know data will be represented in that particular way.

There aren't any wide-ranging applications of the technology at the minute, but I can see things like standardised document formats for word processors, in fact anything where you might like to transfer data between two different systems, benefitting down the line.

Like when you wanted to tranfer a document of some form across from two versions of Word, or from WordPro or something, RTF was the 'lowest common denominator', XML will supercede it as the lowest structured form of data representation.

[queball]

Originally posted by Gayle29uk

Note that only steps 1 and 2 change for a new game (step 1 being a single or at most 2 lines of a bash script anyway), steps 3 and 4 will not change by even one line of code.

This is a false comparison. In neither approach do steps 3 and 4 have to change.

PHP is a dynamic language; if you have space in some table with the location of an XML file you could instead store the location of a PHP script, or even the PHP script itself. For example, vBulletin styles have pages that are evaluted with various variables holding the content: the posts and usernames, or whatever. This could instead be done by having each template as a seperate process reading in the posts' content in XML format and writing out HTML, but that would be horrendous over-engineering.

Clearly there is no harm in doing anything creative. I just worry that you could be doing something more fun; an XML API is hardly interesting. OTOH you should be able to use XSLT to do the pre-parser job with dumpfiles already in XML such as SS or DoM which could certainly be interesting, particularly if you are new to declarative languages.

So my advice would be have a simple PHP API and if the need arises then have an XML interface on top.

And more generally to pab's kind of comments: don't obey the whims of imaginary users. If someone wanted an XML API in anything I write they could send me an e-mail and have it in ten minutes. Or if I don't have the time I could ask for money, but I'd never do anything for imaginary people. Unless you can yourself see a need for XML, do something more interesting.

[Add]

Originally posted by queball
What's the actual advantage of using an external format (XML) instead of an internal one (an API)? Are you saying it's an advantage that the dump translators and the site code are seperate processes/codebases/whatever?

i have no idea. i have never used XML and i doubt i will find any practical use for it in the future. in my post i was simply stating for the poster above what the percieved advantages were. if i were gayle i wouldn't waste my time - she already has a site which pulls stuff from a db, all she really needs is a parser that parses db dumps from other games into her already established db, then her site will (should) work with no recoding. That is assuming her site is structured like that.

[W]

Originally posted by Add
i assume, the tools site would read a standard format of xml. When Gayle wants to add a new game, i again assume, she would write a 'pre-'parser to parse it's dump into her standard format, then simply 'load' the xml file into the tools site config and all of the information would be updated for the new game.

And I ask again, how does this save work? You won't have to code a new parser for each game, but you have to code a new pre-parser. You don't have to code the frontend for each game, but you have to modify the old one. I don't see how this can possibly save time.

[SYMM]

With no clue whatsoever about XML, I think its a mainly theoretical thing, that if there were a lot of games (i.e. more than PA and DoM) then it would save work. Otherwise, I can't see the point either...

[Add]

Originally posted by W
And I ask again, how does this save work? You won't have to code a new parser for each game, but you have to code a new pre-parser. You don't have to code the frontend for each game, but you have to modify the old one. I don't see how this can possibly save time.

if the front-end were properly coded it wouldn't actually need to be changed. I think that is where the work would be saved. personally i don't think it's worth it. that's just me.

[wu_trax]

i would guess gayle doesnt want to use xml because of the higher efficency but because she wants to try out some of this xml-stuff.
(and yes, the idea is probably to avoid haveing to change the code of the real application, but instead write a new stylesheet or parser or whatever)

[pablissimo]

Originally posted by W
And I ask again, how does this save work? You won't have to code a new parser for each game, but you have to code a new pre-parser. You don't have to code the frontend for each game, but you have to modify the old one. I don't see how this can possibly save time.

You don't need to code a new front-end for the new game surely, as all the field names aand general structure is defined in the XML file?

Maybe I'm misinterpreting what is being done here.

[W]

Originally posted by Add
if the front-end were properly coded it wouldn't actually need to be changed. I think that is where the work would be saved. personally i don't think it's worth it. that's just me.

I don't see how you could NOT have to change it, assuming you wanted more than just "sort by" and "search by" the various fields, and some general statistics...

[Add]

Originally posted by W
I don't see how you could NOT have to change it, assuming you wanted more than just "sort by" and "search by" the various fields, and some general statistics...

as far as i know, that's all she needs.

[virogenesis]

why not try it as access dumps out xml files so open up the dumps in access then just use the save as fuction & some1 has asked on google groups but using pearl i believe it was.

[MT]

Originally posted by pablissimo
It's not meant for human digestion though.

It's meant to be easily parsable and structured in a way that can represent tabular data.

And it does this well.

I think you need to look up the word 'easily' in the dictionary. A table of data with fields seperated by some special char and rows seperated by another is much easier to parse. The only advantage XML brings is in describing the data for some remote service that may not know about the data. In this case, Gayle's preprocessor is transforming many different kinds of data into one final data form. Why this needs to be in XML is not apparent.

[MT]

Also libexpat is fking slow. In fact, the only thing slower than libexpat is sablotron.

[Gayle29uk]

Originally posted by MT
In this case, Gayle's preprocessor is transforming many different kinds of data into one final data form. Why this needs to be in XML is not apparent.

Further thought has brought about the conlusion that a final format that mysql can understand through LOAD DATA INFILE would be more sensible

[pablissimo]

Originally posted by MT
I think you need to look up the word 'easily' in the dictionary. A table of data with fields seperated by some special char and rows seperated by another is much easier to parse. The only advantage XML brings is in describing the data for some remote service that may not know about the data. In this case, Gayle's preprocessor is transforming many different kinds of data into one final data form. Why this needs to be in XML is not apparent.

I was referring to the fact that there are pre-built functions for loading and parsing XML that make the process 'easy' whereas creating her own custom data format would have required her to write routines herself.

Which is easier?

[MT]

iirc you still need to write parser functions to actually deal with the xml, then do inserts to actually put the data in..

or one fkoff quick load data..