EVE Technology Lab

In terms of the config file, if you are writing your uploader in C/C++, i'd recommend iniparser. I use it in my own apps and it is a wonderful pre-made solution. Just download the files, grab out the 4 code files (2.c, 2.h) and include them. It works great.

http://ndevilla.free.fr/iniparser/

If you need an XML parser, I've been using a customized version of http://www.applied-mathematics.net/tools/xmlParser.html this guy's parser for several years now. It is fast, correct, etc. I like it. It isn't a library, just like iniparser. You just take the .h and .c files and compile with them.

What did I tweak in his code? Well, mainly XMLParser decided that if *anything* went wrong, he should exit your application. Yea...no. I went through and stripped all that garbage out and replaced it with more graceful failures my app can catch and deal with. I think I might have changed a couple other things, but that is the big one that I remember annoying me for weeks on end.

Other than that...*shrug* sit down and code. :)

It was processing the queue, but not fast enough. That is, I had had the EVE Central import turned off for a while, and when I enabled it again it immediately resumed processing where it left off, filling the queue with about 2 weeks of stale data, which I only just removed in a controlled way.



Interesting to see this uptime, even if it doesn't show EVE Marketeer in a very good light it is probably correct. Even if the website is cool, it suffers from a lot of problems and it is mostly in the experimental stage right now.

Quick question regarding the EVE Marketeer Uploader, does it send all the data directly to each of the sites or does it queue up locally on your server before being sent to the other sites? Not that I have any clue whatsoever about programming, I'm just curious.

LOL, not cheerfully. Anyway, the 100% for Eve Central may not be entirely accurate. When it's not working, it returns a proxy error page and the monitoring system is probably counting this as being a response on port 80, even though the site isn't providing data.

Callean, I'm not sure if you've been working on the server to improve stability, but if you have, it's working.

Eve Central: 100% uptime
Eve Market Data: 95.9% uptime
Eve Marketeer: 92.1% uptime

I was going to suggest something like RSS with JSON payload but after looking into RSS a little more it seems it can't handle what is needed so maybe just straight forward RESTful web application type interface might be the way to go. The format I suggested back in post #36 might be usable directly or with a few changes.

I've also looked at adding the same thing to Yapeal to grab market order information but some other things have had a higher priority lately (SkyRim) for me

The idea with developer_key was to have some kind of site key that others could check mostly just to have an idea where it came from to prevent loops. It could probably be better named. I thought about having some kind of two part encryption system but then we'd have to deal with the whole trust issue and I thought it would be better let each site decide how they wished to handle that part but I think there needs to be some way inside the meta data to track at least where it says it came from.

Ok, I have taken the EMK format, massaged it a bit, and tossed it up on the EVE-Central Development Wiki (hint: anyone can edit this).

http://dev.eve-central.com/unifieduploader/start

Comments?

At present (or for at least a year) I am not going to deprecate the old "throw CSV at server" upload system in EVE-Central, since there are now multiple clients which reference it.

@Kaladr Linky no worky for me. It just times out but even www.eve-central.com is being really slow right now too and is probably going to time out.

Like your ideas but I think it would be a good idea to still do the data part with the extra header stuff from my improved version.

"log":{
"header":["order_id","buysell","solar_system_id","station_id","price",
"vol_entered","vol_remaining","min_volume","issue_datetime","duration",
"range"],
"rows":[
[...]
]

It would make it more self documenting which would allow developers that weren't involved in this thread to still understand what everything was and also makes any future changes easier if needed to add additional columns etc. It would also let developers have more freedom if needed in the order of the columns should it be ever be required in their application. And then there is of course the most important reason it makes it easier for anyone (read me) when I'm trying to fix something that broke at 3am and I haven't had any sleep for 3 days so it's hard for me to remember which came first vol_entered or vol_remaining etc The extra overhead is minimal but adds a lot of value IMHO.

I do like the idea of maybe using UUIDs but how would it be better than just using a MD5 or SHA1 hash of the URL with parameters? Also which parameters were you thinking about using/including in it?

BTW thanks for fixing it so I could see the linky