EVE Technology Lab

This structure does save a lot of overhead but if anyone can really show where the versatility of the other way would be needed we can look at it. The other thing is the 'resultType' would have to be moved into the rowsets as well to mix order and history together which no one seems to have asked for before.

I just didn't see anyone really needing to mix order and history in the same message anyway. To go along with that there's nothing stopping you from including more than one message in a response as long as both parties want to do that and actually that will probably happen in the case of the pushes since it makes sense to just make a single connection to the receiver and send all the data to them when you can. I even see a case for the uploaders to make a connection and keep it open and send the scan results without having to reconnect all the time.

Thought I'd bump this back up to the top again before it got lost.

Mysql is pretty poor on scaling.


Postgres is better, though you'll want some form of connection pooling.

If you are using mysql I'd suggest making sure you have result caching turned on, and batch your updates (so you don't constantly invalidate the cache)

Using tokuDb instead of normal MyISAM or InnoDB tables is a great idea if you are dealing with really large tables.
http://www.tokutek.com/
I've done some testing even on fairly small data sets and it's much better there too.

I also prefer to use MariaDB over normal MySQL as it seems to work better and has less bugs in the main code.
http://mariadb.org/

Everyone I know of that uses Yapeal in large deployments runs into the same problems with network I/O (rarely), DB bandwidth which when you look into it in depth turns out to be Disk I/O problems 99.9% of the time. Most shared hosting has very poor disk I/O so lots of memory to handle surges is a must but at some point it has to go to disk and then your 3GHz 8 way server just became a 50Mbps server in most cases

EC does the "buy/sell" trade calculation in a (very unmaintainable) SQL query. The path finding is farmed out to C++, and the whole deal is sorted in Python.

The "sell/sell" trade finder is managed in Python.

The new buy/sell trade finder is now in Scala similar to how the sell/sell one worked before. Once Hotspot has had a pass at it its nice and speedy.

Just thought it would be a good idea to bump this back to the top again

issuedDate in the EVE cache is actually a strange windows time format, that absolutely has to be parsed, as far as I have been able to understand.

At least, I assume that is the one you are talking about.

Sure, as far as I know, but why give the server even more to do when it can just as easily be done on the client, and leave you with a more readable format when sending? I know I'd much rather see a real date than a string of numbers.

EDIT: It just occurred to me that parsing a datetime string is probably more of a hurdle for the server, but I still vote that over an esoteric numeric format, that everyone will think is a timestamp, while it really isn't.

This was discussed some where earlier in the thread but I'm not finding it right now. The reason we choose the current format is it is easy to understand for people and computers. It's not OS dependent and most databases can work with it directly or it can be translate easily for those that don't. Most programming languages also can handle it with little trouble when need and more important since we're developing a 'standard' it makes much more sense to leverage other existing ones that have been well thought out, peer reviewed, and is well documented.

One thing to keep in mind is this is a data exchange format and not a data storage format and as originally stated we wanted it to be understandable even by a programmer that had never seen it before without having to read a bunch of docs somewhere. Much like JSON itself being able to describe it with a single page and just a couple of examples with descriptions was at least part of my original concept and I believe everyone else's. We also wanted something that was compact but not to the point that it becomes hard to understand or use. There was some early talk about using XML but it's just to bloated for something like this so that's where JSON won out.