These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

RETURNED - EVE Marketeer - Market Data Aggregation & Information

Author
Vaerah Vahrokha
Vahrokh Consulting
#521 - 2012-03-30 23:21:54 UTC
More than 500k uploads backlog, please unlock it.
Scrapyard Bob
EVE University
Ivy League
#522 - 2012-03-30 23:41:38 UTC
Fix it fix it fix it?
http://www.youtube.com/watch?v=1Isjgc0oX0s

540k backog now.
Callean Drevus
Perkone
Caldari State
#523 - 2012-04-01 10:24:34 UTC  |  Edited by: Callean Drevus
Sorry guys, after fanfest I was (and am, probably) suffering from EVE overload, and I haven't been looking at the website for a bit, except to see whether it was running.

My ISP informed me that they were going to restart one of my servers to do some maintenance, but unfortunately informed me wrongly Roll, and restarted the EMK server without me knowning, leading to the EMK processes stopping and not being started again for a few days. The maintenance is now complete, and the uploads are being processed again.

Should be steadily decreasing the backlog now. Last time I checked about 50000 / 2.5 hours.

UPDATE: Nothing has changed to the market scanner in the past weeks... I cannot imagine any reason for the errors except a change in the client. Which would be odd, since the other market sites are obviously working...

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#524 - 2012-04-02 13:29:38 UTC
Thinking about how to calculate the top-10 market hubs again.

The naive method - just sum up all the buy orders value in that location, report on that. Then sum up all the sell order value, report on that. Maybe drop the outliers first.

The 5% method - Take the 5% sell price for each item, multiple by the total number of units for sale. This avoids the "1M ISK shuttle" from skewing the numbers. May as well do the same for the buy price.

Sum of the daily transactions - Use the history data, average the past 7 or 14 days, sum up the amount of ISK that changed hands. But this is only region-wide, not station/system.

Things I would like to see:

- Top market regions by ISK value, both top 10 buy-order regions, top 10 sell-order regions and top 10 combined value.

- Top systems by ISK value (so all of the stations in Jita, not just CNAP 4-4).

- Top stations by ISK value

You should track the top 20 locations in a table each day so you can present the top 5 or top 10 as a graph showing how they stack up over time.

I'm very curious to see how the various regions stack up against each other, and what the top 10 regions / systems are.
Vaerah Vahrokha
Vahrokh Consulting
#525 - 2012-04-03 10:14:11 UTC
@Scrapyard

I noticed the backend used for this website has seems to have poor concurrency locking granularity. When you send some large queries (aggregates like the ones you wish above) it may take several seconds. Some database back ends lock the tables during those queries and therefore somebody trying to upload their data will be affected.
Callean Drevus
Perkone
Caldari State
#526 - 2012-04-03 16:16:36 UTC
The uploading process is pretty much unaffected by big queries (at least, as far a mysql is concerned), but the server itself might be a bit slow. Regardless, this is a query you will do at most once a day, and cache the results. That is acceptable in my opinion.

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#527 - 2012-04-04 05:37:45 UTC
Speaking of big, slow queries... the Consumer Price Index page takes forever.

http://www.evemarketeer.com/trade/consumerindex
Scrapyard Bob
EVE University
Ivy League
#528 - 2012-04-05 00:52:37 UTC
So, if I go to this link:
http://www.evemarketeer.com/uploader

The "play" button only works about 1/2 the time at getting the market window to open up.

But if I go to:
http://www.evemarketeer.com/uploader/display/all/orders

Then it mostly seems to work correctly.

When it breaks, it simply won't open the market window, but the timer sits and cycles through items and counts down.
Scrapyard Bob
EVE University
Ivy League
#529 - 2012-04-05 13:13:11 UTC
http://www.evemarketeer.com/statistics/uploads

Uploads are backed up again (129k and counting).
Callean Drevus
Perkone
Caldari State
#530 - 2012-04-07 09:49:29 UTC  |  Edited by: Callean Drevus
I finally found time to do something with EMK again, so I'll be looking into this today. And see if I cannot have those processes autostart.

Update: Going to http://www.evemarketeer.com/uploader and clicking play, works for me each and every time. Is there anything specific you are doing before that? I just open the browser and go right to that page then click play.

In addition on what machine are you running EVE, and what are it's settings?

Just trying to gather some information :)

Developer/Creator of EVE Marketeer

Scrapyard Bob
EVE University
Ivy League
#531 - 2012-04-07 11:44:43 UTC
Callean Drevus wrote:

Update: Going to http://www.evemarketeer.com/uploader and clicking play, works for me each and every time. Is there anything specific you are doing before that? I just open the browser and go right to that page then click play.

In addition on what machine are you running EVE, and what are it's settings?


Running in 1024x768 minimum setting windowed clients. Alt-B, EMK's market scanner is the home page, click play. It just seems like the countdown timer is a free running bicycle chain which never "catches" to open up the market window. Once it does latch on, it works fine.

EMD's page works flawlessly every time.

My guess, opening up the window causes a EVE API call, and clicking play button too soon might result in a second EVE API call before the 3-second timer resulting in a mute.
Vaerah Vahrokha
Vahrokh Consulting
#532 - 2012-04-08 21:23:01 UTC
Since the other day when the upload stopped working, I am unable to send prices any more.
I have cache files delete enabled on startup. It deletes them but won't upload anything, I just see the windows open and close like in the past but nothing gets sent.
Scrapyard Bob
EVE University
Ivy League
#533 - 2012-04-09 13:07:22 UTC
Vaerah Vahrokha wrote:
Since the other day when the upload stopped working, I am unable to send prices any more.
I have cache files delete enabled on startup. It deletes them but won't upload anything, I just see the windows open and close like in the past but nothing gets sent.


Check to see if it lost track of your upload keys.
Try clearing cache for each junction point.
If you have multiple junction points, make sure you fire up a EVE instance for each junction point (fixes the cache).
Restart the uploader.

(The above works for me 99.5% of the time.)
Vaerah Vahrokha
Vahrokh Consulting
#534 - 2012-04-09 21:48:23 UTC
Scrapyard Bob wrote:
Vaerah Vahrokha wrote:
Since the other day when the upload stopped working, I am unable to send prices any more.
I have cache files delete enabled on startup. It deletes them but won't upload anything, I just see the windows open and close like in the past but nothing gets sent.


Check to see if it lost track of your upload keys.
Try clearing cache for each junction point.
If you have multiple junction points, make sure you fire up a EVE instance for each junction point (fixes the cache).
Restart the uploader.

(The above works for me 99.5 of the time.)


I don't have junction points, I have a second computer I put only to do this sort of stuff (only 1 client ever runs in there).

I noticed on my main computer I can get never-upload-before items do get uploaded (the others won't, due to an issue Callean told earlier).

On the second computer it just won't upload anything ever.
If I run my own made cache reader I can pull out everything easy, as this post can prove. The fun thing is, that second computer used to be the one that flawlessy sent everything up.

Is it possible that a recent patch created another "machonet" folder so the EvE Marketeer uploader does not delete the old folder files?
If so, what should I manually delete to be sure I don't bork something like i.e. my settings?
Scrapyard Bob
EVE University
Ivy League
#535 - 2012-04-10 00:28:08 UTC
Vaerah Vahrokha wrote:

If so, what should I manually delete to be sure I don't bork something like i.e. my settings?


Settings files on Win7 are under:

C:\Users\username\AppData\Local\CCP\EVE\c_program_files_(x86)_ccp_eve_tranquility\settings

You should always backup the settings folder regularly.

Cache folders are in the same directory tree and can be freely deleted (if the EVE clients are shutdown).

If you run junctions or multiple clients, then you'll have multiple folders under CCP\EVE, one for each path to the EVE executable.
Gath Ra'nan
Brutor Tribe
Minmatar Republic
#536 - 2012-04-10 09:42:32 UTC  |  Edited by: Gath Ra'nan
FYI,

Fatal error: Uncaught exception 'Predis_CommunicationException' with message 'Connection refused' in /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php:1520 Stack trace: #0 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(1470): Predis_Connection->onCommunicationException('Connection refu...', 111) #1 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(1446): Predis_Connection->redisStreamInitializer(Object(Predis_ConnectionParameters)) #2 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(1600): Predis_Connection->connect() #3 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(1551): Predis_Connection->getSocket() #4 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(1525): Predis_Connection->writeBytes('*2??$3??GET??$1...') #5 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(1534): Predis_Connection->writeCommand(Object(Predis_Commands_Get)) #6 /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php(203): Predis_Connection->executeCommand(Object(Predi in /var/www/www.evemarketeer.com/core/lib/redis/lib/Predis.php on line 1520

Great site, btw
Callean Drevus
Perkone
Caldari State
#537 - 2012-04-10 14:39:30 UTC  |  Edited by: Callean Drevus
Oh joy...

I'm starting to hate the server maintenance that seem to have to happen every week... It is fixed. Uploads will not be processed though, until I can restart those processes.

Regarding the uploader, it should always use the newest cache folder on startup, an completely ignore any others.

Developer/Creator of EVE Marketeer

Styth spiting
Brutor Tribe
Minmatar Republic
#538 - 2012-04-10 15:52:57 UTC
I wanted to find out if you are interested in another developer getting involved in this project. I've all ready spent roughly 40 - 60 hours working with the code and have a lot of ideas for new features (and performance ideas).

The problem I am running into though is that there is no data dump of default values (items for example) so even after getting everything setup on my dev server I cannot duplicate the existing site. Its basically a functioning site with no data (pretty important).

Some of the things I have been working on are:


  1. consolidating many of the configs into 1 primary config file.
  2. Installation instructions (mainly for myself, but available to everyone).
  3. Properly sized icons (image icons are way to big for display withing the sites layout).
  4. Item database. Adding aucutal item data, not just item market data.
  5. Items reprocessed amounts.
  6. css / html optimization.
  7. various calculators and tools for players to use (ore to mineral converter using market data for example.


So yeah, if you want a kick ass developer helping out send me a message and we can get some discussion going. I love the project (and the existing code is pretty sexy) and would love to get involved with it.
Snarf Aldes
University of Caille
Gallente Federation
#539 - 2012-04-10 18:46:28 UTC
I think i found an issue with the unified uploader.

It seems that if one of the endpoints has issues, and the queue fills to 50, uploads to the properly working endpoints also stalls.
It looks like uploads only happen as long as none of the queues are filled to 50.

Creator of Eve Addicts

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#540 - 2012-04-13 16:53:16 UTC
This may be related to the uploads not getting processed, but this appears to be returning nothing:

http://www.evemarketeer.com/api/recent_updates/5/xml

If uploads still aren't processing, that would make sense. But it is a bummer for me, trying to steal your data :)

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com