These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information
Author
Previous Topic Next Topic
Scrapyard Bob
EVE University
Ivy League
#241 - 2011-12-24 13:54:53 UTC
I figured out a new way to break the API call by putting garbage in for the type IDs.

http://www.evemarketeer.com/api/info/32_#N/A_33_#N/A_#N/A_#N/A_#N/A_#N/A_#N/A_#N/A/xml/10000002/sell_lowest5

This results in the error:

MYSQL ERROR OCCURRED: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') AND station_id = 0 ORDER BY FIELD(type_id, 32,)' at line 1

Which may possibly mean that you are vulnerable to a SQL injection. You'll need to ensure that those values are forced into numeric types before passing them to the SQL engine.
Callean Drevus
Perkone
Caldari State
#242 - 2011-12-25 08:56:12 UTC  |  Edited by: Callean Drevus
They are escaped before adding them into the query, that said, it's a bit weird they actually break it, but probably because the query expects them all to be numbers.

Thad said, I"ve added it to the feedback page and will see if I can fix it.

As for the Win XP error. I guess that's what I get for trying to lend stuff from EMD, the odd part is that it's actually designed to do that differently for XP, so I guessed it should work. Will fix in next version.

Developer/Creator of EVE Marketeer
Invictra Atreides
Toward the Terra
#243 - 2011-12-27 01:44:53 UTC
Today I just started to use your uploader and the XML API. Now when I run the uploader it does its thing and uploads the files, but when I check on evemarketeer.com the data is not updated. It still shows the old data. why is that ?

It also uploads to eve-central.com so I also checked there and the data there was updated.

Then I also checked eve-marketdata.com and the data there was also updated, but their XML API was still showing the old data.

Is this a normal behavior? I don't understand.

Blog | Tutorials | Youtube "I don’t know everything, I just know what I know."
Callean Drevus
Perkone
Caldari State
#244 - 2011-12-27 01:52:46 UTC
Invictra Atreides wrote:
Is this a normal behavior? I don't understand.


It both is, and is not at the same time. The server was rebooted yesterday, and wasn't running any processing threads due to them all being killed because of that. It should be updating normally now. Anyhow, I really need to start doing this automatically, as these are situations I like to avoid, so hopefully this problem won't occur again.

If you ever wonder whether anything is happening to your updates, you can look at the upload statistics page. If the EVE Marketeer backlog counter keeps increasing, something might be wrong with the processing (but your uploads are still saved, and will be processed when the problem is resolved).

Developer/Creator of EVE Marketeer
Scrapyard Bob
EVE University
Ivy League
#245 - 2011-12-27 14:49:46 UTC
Invictra Atreides
Toward the Terra
#246 - 2011-12-27 21:24:00 UTC
Scrapyard Bob wrote:
http://www.evemarketeer.com/api/info/34_35_36_37_38_39_40_11399/xml/10000002/sell_lowest5

Is showing zero data for Pyerite, Mexallon and Zydrine this morning.

http://www.evemarketeer.com/item/info/pyerite
http://www.evemarketeer.com/item/info/zydrine
http://www.evemarketeer.com/item/info/mexallon
It seems half the database was corrupted. The restore data on the webpage helps.

Blog | Tutorials | Youtube "I don’t know everything, I just know what I know."
Callean Drevus
Perkone
Caldari State
#247 - 2011-12-28 10:13:12 UTC  |  Edited by: Callean Drevus
Well, it isn't that the database is corrupted, it is more that EVE Central keeps sending empty data at times. I thought I'd fixed the problem where empty EVE Central data was sent, but it seems not. I built in the restore button in any case, so you can go back to correct data if you think the data can't possibly be right.

I'll see what is happening to make some data still being cleared.

UPDATE: Uh... here I was badmouthing EVE Central, when it was really a failure of my own. It'll be fixed in no time.

Developer/Creator of EVE Marketeer
TheSmokingHertog
Julia's Interstellar Trade Emperium
#248 - 2011-12-28 11:16:55 UTC
Just found out about your site / tools. Thx for all efforts. Lets see how to put the information to good use over here :).

"Dogma is kind of like quantum physics, observing the dogma state will change it." ~ CCP Prism X

"Schrödinger's Missile. I dig it." ~ Makari Aeron

-= "Brain in a Box on Singularity" - April 2015 =-
Kaedaar OLeary
ADK Industries
#249 - 2011-12-29 01:48:15 UTC
Any idea how I can upload to the battleclinic servers as well. I see you can add more endpoints in your uploader.


Thanks for all your efforts.
Callean Drevus
Perkone
Caldari State
#250 - 2011-12-29 09:36:32 UTC
I am not aware of battleclinic supporting direct uploading. If and when they implement one of the upload formats, they will publish an endpoint url that you can add to the uploader.

Developer/Creator of EVE Marketeer
Rektide
School of Applied Knowledge
Caldari State
#251 - 2011-12-29 17:54:39 UTC
Hi. I'd like to request some kind of streaming output. EVE Metrics had an AMQP feed, EVE Central has an email list that sends out an email with all of the raw CSV received in the last minute: I'd love something with that push-based streaming out in EVE Marketteer.

There is recent_updates, but it's something that needs to be polled & updated manually. I'm also afraid of missing contents, so I'd have to request time_since_last+1 then dedupe, which adds a little cost & complexity.

Connectivity options: email, amqp, webhook callbacks, xmpp MUC bot, zeromq, ...
Dragonaire
Here there be Dragons
#252 - 2011-12-30 04:43:19 UTC
Might check out this thread as there are plans to have a unified exchange format that is being developed that all the upload sites and others are working on implementing including EMK
https://forums.eveonline.com/default.aspx?g=posts&t=18466&find=unread

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Scrapyard Bob
EVE University
Ivy League
#253 - 2012-01-01 01:51:14 UTC
On the market scanner page where it shows "(6857 / 6922 updated in last 24 hours)" - that number does not appear to be updating as time goes on. So, for instance if it shows 12% has been updated in the last 24 hours, then you look at the regional page it will say something far higher after a few hours.

Also, the regional status page does not always render the first time visited in Firefox. I have to go to that URL, then force a page reload in order for it to display properly. It's a hit/miss error, possibly cache related. It may only happen once per hour (as the page gets updated).

http://www.evemarketeer.com/statistics/regions

PS: The unified uploader is running happily, 174k cache files processed since it started and no crashes. There does seem to be a memory leak somewhere because it's up to 257MB of "peak working memory" with 183MB in current use (174MB private/commit size). It started at around 35-40MB when first launched.
Callean Drevus
Perkone
Caldari State
#254 - 2012-01-01 18:11:35 UTC  |  Edited by: Callean Drevus
@Rektide: There are still plans for streaming output using AMQP, but at the moment it's not very high on my priority list. I might be able to simulate something EVE Central like for the interval though.

@Scrapyard: I know, there is something going wrong with the way those values are calculated. I'll investigate the issue. Same goes for regions, though I have no clue as to what is causing that yet.

Good to hear that the unified uploader is running happily. I wonder what is causing the large memory consumption though. How long have you had it running?

UPDATE: I've finally gotten so far that I've added features that are only accessible to people with upload credits (or donation credits, but nobody ever did that, and they're worth less than upload credits), that can be used to do a variety of things (of course it won't have any effect on the top uploaders page). The first one of these is adding images to the homepage, that will be shown for a week before disappearing (or being extended of course). Take a look at the new option under 'My Market'.

Developer/Creator of EVE Marketeer
Scrapyard Bob
EVE University
Ivy League
#255 - 2012-01-01 23:12:14 UTC
Callean Drevus wrote:

Good to hear that the unified uploader is running happily. I wonder what is causing the large memory consumption though. How long have you had it running?


A a guess, somewhere around 9 days and 174,000 cache files processed. So it's a fairly slow memory leak. I have 16GB of RAM, but I could see that being an issue for others.

The ".log" files don't put time-stamps in when they start-up (it just says "Logger started").
Callean Drevus
Perkone
Caldari State
#256 - 2012-01-03 23:17:17 UTC  |  Edited by: Callean Drevus
Ok, I guess that means it's not a very radical problem, and it explains why it hasn't shown up in my tests. Will still see whether I can do something about that, but my guess is most people don't leave their computers on for 9 days ;)

In addition, if any of you ever wondered what your wallet balance graph looks like, which I guess you didn't since we all know of EVE Mentat and all those programs, I've always felt that such a function was missing on EVE Marketeer, so I've added it.

Find it under My Market -> Wallet History

Developer/Creator of EVE Marketeer
Scrapyard Bob
EVE University
Ivy League
#257 - 2012-01-04 08:13:49 UTC
And a small bump because the history tab is showing chart history that only goes through Nov 27th.

http://www.evemarketeer.com/item/info/pyerite

(It's probably already on your to-do list.)
Callean Drevus
Perkone
Caldari State
#258 - 2012-01-04 10:38:44 UTC  |  Edited by: Callean Drevus
And you did upload more history than November I take it? :P It wasn't on my to do list yet Blink thanks for letting me know.

UPDATE: Stupid error had to do with the Unified Uploader not uploading history at all, and even if it did, uploading it incorrectly. This has been fixed now. A new version (0.4b) has been uploaded to the website.

Which means it was somewhat on my to do list, since I knew I hadn't tested history uploading with the uploader yet. Anyway, it works now :)

Developer/Creator of EVE Marketeer
Scrapyard Bob
EVE University
Ivy League
#259 - 2012-01-04 14:18:15 UTC  |  Edited by: Scrapyard Bob
Which reminds me of a question. For eve-marketdata, you had to manually switch your market screen to be looking at the "history" tab in order for it to gather data before running scans. Is that still true that the market history cache files will not be generated if you do not specifically switch to the "history" tab?

Edit: Given that, if I browse the market with the "history" tab open - eve-central receives no updated data, I'm guessing that the answer is "yes"? In which case there needs to be some sort of text on the EMK market-scanner page to remind people that they should sometimes do market scans with the "price history" tab open.
Kaladr
Viziam
Amarr Empire
#260 - 2012-01-05 00:32:04 UTC  |  Edited by: Kaladr
Yes, the history tab needs to be open for the client to make calls to get history (and in turn end up in the cache files).

EVEC still has no history interface, so of course no uploads go to it. Maybe soon, but there is a lot of core stuff in the primary "orders" land which is still taking up my time.

Creator of EVE-Central.com, the longest running EVE Market Aggregator