These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Why are we not combining market data, player devs?
Author
Previous Topic Next Topic
Dragonaire
Here there be Dragons
#181 - 2012-03-21 17:24:18 UTC  |  Edited by: Dragonaire
One thing that has stalled this whole idea is none of the upload sites accepting the new format yet as far as I know which is also part of my reason for bumping this thread as a small hint to those developers they need to implement this like they all said they would Blink I've seen lots of sites jumping on the universal uploader stuff but except if I missed the change it was still using the old format which has some know problems Blink

I even started a project on Source Forge where people could start adding example code/libraries in different languages so everybody wasn't going end up reinventing the wheel Blink
https://sourceforge.net/p/evemarketfeed/home/Home/
but I haven't had anyone contact me about adding anything Sad

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Callean Drevus
Perkone
Caldari State
#182 - 2012-03-21 17:30:51 UTC
Hehe, this is true, I also need to get working on this. But I also need to get working on a lot of other stuff ;)

Luckily, implementing the new upload format should not be too difficult.

Developer/Creator of EVE Marketeer
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#183 - 2012-03-23 03:18:50 UTC
Is this up to date with all of the latest changes? http://dev.eve-central.com/unifieduploader/start

I'm working on a project that is very relevant to this thread that may actually see the light of day.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Dragonaire
Here there be Dragons
#184 - 2012-03-23 23:05:31 UTC
Yes it should be up to date as there real has been any changes to it for last couple months. It also has everything needed by all the developers that gave input on it. It's probably safe to say this is at least the Beta or first RC version of it and until it's starts being used there's no known bugs at this point Blink

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Kaara Jade
The Scope
Gallente Federation
#185 - 2012-03-26 06:22:17 UTC
Hi, I'm coming into this thread and trying to make heads or tails of what the current state is and what the intended use is.

The docs,
http://dev.eve-central.com/unifieduploader/start

are fairly unclear. There are only two endpoints, upload and syndication, ok. How are the message formats, orders and history, related to these two endpoints? I'd really like to see some round trip examples of request/response to and from, demonstrating the use cases intended. How is syndication supposed to work?

I'd much prefer some example HTTP request/responses demonstrating use cases than any words describing the intent.
Snarf Aldes
University of Caille
Gallente Federation
#186 - 2012-03-27 06:56:38 UTC
Kaladr wrote:


Some of it is PostgreSQL, some of it is heavy-handed caching, some of it is luck (maybe? Big smile)

Honestly, the biggest limit is disk bandwidth, which is a limit for writes (and impacts reads). You can cheat and run PostgreSQL in "NoSQL mode" (turn off synchronous commits), which erodes at the durability/server shutdown survivability.

The EC active dataset can be cached in RAM fairly easily (which is why I run with no less than 16GB, and the new system runs at 48GB), which helps read performance. Believe it or not, I am actually running into CPU contention now...

Out of curiosity, how many uploads does EC get at peak hours (in uploads per minute or something similar)?

Same question for EMK...

It must be WAY more than what i'm getting now, peak was about 180 uploads per minute.

Creator of Eve Addicts
Snarf Aldes
University of Caille
Gallente Federation
#187 - 2012-03-27 09:05:00 UTC
Kaara Jade wrote:
Hi, I'm coming into this thread and trying to make heads or tails of what the current state is and what the intended use is.

The docs,
http://dev.eve-central.com/unifieduploader/start

are fairly unclear. There are only two endpoints, upload and syndication, ok. How are the message formats, orders and history, related to these two endpoints? I'd really like to see some round trip examples of request/response to and from, demonstrating the use cases intended. How is syndication supposed to work?

I'd much prefer some example HTTP request/responses demonstrating use cases than any words describing the intent.

I also have some questions about this format.

In the example there are cases where regionID and / or solarsystemID are 'null', why?
In my opinion this should never happen, the uploading client should provide all necessary info.

Creator of Eve Addicts
Dragonaire
Here there be Dragons
#188 - 2012-03-27 17:03:29 UTC
Quote:
I'd much prefer some example HTTP request/responses demonstrating use cases than any words describing the intent.
I think you've missed the point as the actual way the message is sent from the data source to the data sink doesn't matter though the expected sink would be sites like EMK etc but since the messages are also expected to be use for data exchange between sites like it as well. EMK or any of the others can act as sources to each other to help fill in holes in their data from when they are down for maintenance etc. The idea is to do much as killboards do now where they exchange kills to get a more complete picture.
Quote:
In the example there are cases where regionID and / or solarsystemID are 'null', why?
In my opinion this should never happen, the uploading client should provide all necessary info.
That is true if you are only getting your data from the cache but if you are getting your data from the Market APIs etc that information isn't available. There are plans it sounds like for EveMon to act as a data source and I have plans to also add sourcing option from Yapeal and both would be using the API data which only includes the stationID. The idea is to increase the amount of sources where data can come from so more people can help out without having to always run the current cache based tools like the unified uploader. Part of the problem with relying on people running uploaders is they tend to all cover the main hubs only but the API would provide data from any place people have market orders up as long as they use something like EveMon or provide a ApiKey to anything that use Yapeal for their Market API data. There is no reason that other API libraries couldn't add something similar.

I think both of you could benefit from reading the thread start on about page 4 where the formats were hashed out between the developers involved at the time.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Snarf Aldes
University of Caille
Gallente Federation
#189 - 2012-03-28 08:03:21 UTC
Looking up the missing data is of course easy to accomplish, but, i want to keep processing at an absolute minimum.
The hard part for any data aggregation site is the processing / storing of the data. Unfortunately, I don't have the hardware that CCP uses for their database server Roll

It seems to me that all DB maintainers would benefit greatly if the extra processing is done by the client, not the server.

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#190 - 2012-04-03 21:42:02 UTC
Snarf Aldes wrote:
Looking up the missing data is of course easy to accomplish, but, i want to keep processing at an absolute minimum.
The hard part for any data aggregation site is the processing / storing of the data. Unfortunately, I don't have the hardware that CCP uses for their database server Roll

It seems to me that all DB maintainers would benefit greatly if the extra processing is done by the client, not the server.


Stay tuned, I'm working on a project that may alleviate this and other concerns being brought up. It would be premature to announce it just yet before I have time to document it, otherwise people will just discard it.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Scrapyard Bob
EVE University
Ivy League
#191 - 2012-04-04 02:54:13 UTC
Snarf Aldes wrote:

Out of curiosity, how many uploads does EC get at peak hours (in uploads per minute or something similar)?


You can kind of guesstimate it from EMD/EMK numbers. There are roughly 6500 items, and at 3.5 sec/item it takes about 6.3 hours to update the market. So if the "average age" is < 6 hours, then you have multiple people uploading at the same time. In the case of the Forge at the moment, average age is about 2 hours, so there's probably 3 people doing constant uploads. Heim is running at about a 4-hour average so you can estimate that as two people running non-stop.

The Forge = 3
Metro = 1
Heim = 2
Black Rise = 0.5
Derelik = 1
Domain = 1
Kador = 2
Tash-Murkon = 1
Citadel = 0.5
Sinq = 2

Which would mean there are about 14 people total doing constant uploads to EVE-MarketData (EMD). Or about 240/min at peak, maybe a bit more. This estimate matches up fairly well with the EMD upload graph (which is about 200k uploads/day).

We need more people running the upload tool. Especially in the secondary markets such as Everyshore. Also, Domain / Tash-Murkon is a bit under-served if I don't run my market alt there all day.
Dragonaire
Here there be Dragons
#192 - 2012-04-04 05:22:57 UTC  |  Edited by: Dragonaire
Snarf Aldes wrote:
It seems to me that all DB maintainers would benefit greatly if the extra processing is done by the client, not the server.
But asking every project that could act as an uploader to add everything needed to fill in the data where they don't need it for anything but to make it easier for those trying to run a web site on an underpowered server doesn't make sense either. Blink If I had to add a large part of the SDD and make the ConquerableStationList API required to be on to fill in the holes in the SDD I know that personally as a developer I wouldn't even consider adding it to Yapeal. The web sites already need the SDD etc to display the information to their users for the numbers to names stuff where I don't believe any of the clients do. It just makes more sense to have the web server do the job after receiving the data or at the time they display it to their user IMHO.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Kaladr
Viziam
Amarr Empire
#193 - 2012-04-13 04:10:35 UTC
Snarf Aldes wrote:
Kaladr wrote:


Some of it is PostgreSQL, some of it is heavy-handed caching, some of it is luck (maybe? Big smile)

Honestly, the biggest limit is disk bandwidth, which is a limit for writes (and impacts reads). You can cheat and run PostgreSQL in "NoSQL mode" (turn off synchronous commits), which erodes at the durability/server shutdown survivability.

The EC active dataset can be cached in RAM fairly easily (which is why I run with no less than 16GB, and the new system runs at 48GB), which helps read performance. Believe it or not, I am actually running into CPU contention now...

Out of curiosity, how many uploads does EC get at peak hours (in uploads per minute or something similar)?

Same question for EMK...

It must be WAY more than what i'm getting now, peak was about 180 uploads per minute.


I'd have to do more work to figure out the peaks, but using one random day (Apr 9th), total uploads was 174k.

The new database server is really helping with the load here. Unfortunately, I haven't had a chance to deploy any of the "new" stack yet to utilize the extra power. Thats the reality.

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#194 - 2012-04-16 16:17:25 UTC
EVE Market Data Relay has launched in a beta form, which may or may not interest some of you. The primary thing I want to talk about is an excellent point was brought up today:

https://forums.eveonline.com/default.aspx?g=posts&m=1139822#post1139822

Does this make enough sense to work into the spec? I could submit a pull request for the EVE Unified Uploader client for this, if so. It'd help us cut down on duplicate data, since the uploader supports multiple endpoints, and there's a lot of data sharing going on.

In the case of EMDR, a few of you are helping feed it, but if an uploader was pointed at several of our contributors, we'd end up with a lot of duplicate data. This is not dangerous, and if people are storing data correctly, largely harmless. But it is pretty annoying.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Dragonaire
Here there be Dragons
#195 - 2012-04-16 16:52:17 UTC  |  Edited by: Dragonaire
Why not use the uploadKeys array and just add your serial number or whatever? It's there to be used however you want. If every upload site had something that was added to it by the sources you'd have the same information without having to change anything or optionally you can keep track internally where you received data from and add something to all of your forwards to detect loops.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#196 - 2012-04-16 17:04:23 UTC
I'm not sure I understand, but I don't think that fixes the problem. Let's say Player A uploads to EVE Central and EVE Marketdata. EMDR then picks up the same upload, twice, and the data that was sent via their client to both sites would be identical. Any consumers downstream would get two identical payloads.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#197 - 2012-04-16 17:11:05 UTC
It's exactly the same thing as having multiple uploaders supplying the same data (let's say 1000 players sit in Jita and upload the same data for item X). How do the sites process the duplicates?
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#198 - 2012-04-16 17:18:32 UTC
The end result is similar, it'd just be nice to be able to drop known, for-sure duplicates, instead of unique data that is the same (as is the case with the Jita example). To be clear, neither case constitutes data that is harmful, just annoying. I figured I'd raise the issue to see if there were any others annoyed by it enough to figure something out.

If not, then no big deal. It's less work for me.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Dragonaire
Here there be Dragons
#199 - 2012-04-16 17:28:31 UTC
As Desmont McCallock pointed out it's all the same problem and your site is just going to have to do the duplicate detection. What I'm suggesting is a way to help find duplicates and keep from getting into forwarding loops in cases where you receive data from some site and also feed to it. Each site needs to make sure they track data source and not send the data back to the source but that's outside of what this standard is made to address. It's a data exchange format not a routing protocol.

Would it maybe good for all the sites to come up with a system to make routing tables between themselves probably but that's separate from the data format. I'd be more than happy to help develop something along this line if everyone is interested in it.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
LifeHatesMe
LifeHatesUsAll
#200 - 2012-04-16 17:37:24 UTC
Dragonaire wrote:
As Desmont McCallock pointed out it's all the same problem and your site is just going to have to do the duplicate detection. What I'm suggesting is a way to help find duplicates and keep from getting into forwarding loops in cases where you receive data from some site and also feed to it. Each site needs to make sure they track data source and not send the data back to the source but that's outside of what this standard is made to address. It's a data exchange format not a routing protocol.

Would it maybe good for all the sites to come up with a system to make routing tables between themselves probably but that's separate from the data format. I'd be more than happy to help develop something along this line if everyone is interested in it.
I thought this was discussed previously? And there was no concensous because of how it could be abused? Heck, if EVE could supply us with a accessible unique transaction ID (via Cache, or Market Export), that would be hugely helpful, will they though? Remains to be determined..