These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Dragonaire
Here there be Dragons
#221 - 2012-07-14 05:03:16 UTC
Just to add my 0.02 ISK in here about the generatedAt time. It could also be the time from the API XML if that was the source so keep that in mind before getting to carried away on expecting it to be able to tell you which of two messages was first down to the second or even much below a few minutes. In the other thread that you posted a link to this one and in there was some back and forth about how to do the time and that why we decided on using ISO 8601 and there is also some clarifying explanation at the bottom of the page in the spec on using the complete time format so as to not have any confusion about if it was in local or another time zone. We all felt it was important not to make mistakes that we had all seen when that wasn't done like with the API data/times without any time zone on them etc. You also wondered about the sync of varies sources etc. There can be problems there because in the case of the cache files it could be dependent on the clock of the user uploading the data being set correctly, I believe the Eve client uses the time it gets from the Eve server so it should be accurate but someone that is more familiar with the cache would have to answer that.

So to kind of sum it up the date/time from a single source should be directly comparable but when coming from multiple sources it would take getting to know where they all come up with the date/time reported in generatedAt and insuring in your application you adjust for any differences.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Desmont McCallock
#222 - 2012-07-14 08:40:43 UTC
I'm really curious what impact the combination of EMDR/EVEMon had on the EVE economy. Looking forward to see the next 'Price Indices' report.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#223 - 2012-07-18 12:50:22 UTC
ShoGun/Baptiste has volunteered a second french relay to complement Jognu's. This works out well, as there has been quite a bit of usage on the european relays. For those of you in Europe, you may now use the new relay for additional redundancy, or as your sole data provider.

tcp://relay-eu-france-2.eve-emdr.com:8050

Thanks to ShoGun for volunteering it. Here is the full list of relays:

http://www.eve-emdr.com/en/latest/access.html

Greg

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Shellac Brookdale
Cutting Edge Incorporated
#224 - 2012-07-18 19:25:35 UTC
Ilyk,
Desmont --
Great work so far!

But can we please have something like this for killmails too? Have EVEmon users auto upload killmails into a separate data relay network and enjoy all kinds of new killmail consuming apps making use of this? Making it easier for killboards to sync kms? That would be huge.
Desmont McCallock
#225 - 2012-07-18 21:03:57 UTC  |  Edited by: Desmont McCallock
Looking into the possibility of such a project, I began scanning the cache files (with EVECacheParser) to see if the kill mails info can be retrieved. I discovered that the kill mails are indeed cached under the method 'charMgr.GetRecentShipKillsAndLosses'.

The data that can be retrieved are:
killTime, iskLost, killID, solarSystemID, victimCharacterID, victimCorporationID, victimAllianceID, victimFactionID, victimShipTypeID, victimDamageTaken, finalSecurityStatus, finalCharacterID, finalCorporationID, finalAllianceID, finalFactionID, finalShipTypeID, finalDamageDone, moonID, warID, killBlob (which contains info about the attackers and the items dropped).

Now, if we can agree to a unified format much like we did with the market data, I don't see any problems in creating an EKDR (EVE KillReport Data Relay).

Edit: Prerequisite for this to work is that a player has to open the combat log in the character sheet in order for the cache file to be created, much like it works with the market data.
Shellac Brookdale
Cutting Edge Incorporated
#226 - 2012-07-18 21:57:24 UTC
Wouldn't retrieving the killmails trough the API be an option for clients to push data?
Desmont McCallock
#227 - 2012-07-19 06:56:08 UTC
I thought that you might ask. Killmail API is defected as hell. If you have given to a service (site or client) an API key to retrieve kill mails, you can't retrieve the same info (even with another API key) so to use to another service. Meaning that the killmail API call doesn't return persisting data.

This is also the reason I haven't added a kill mail monitor in EVEMon yet. Those that use any API key to feed a killboard won't be able to use EVEMon (or any other app/service that use the same API call) to monitor the kill mails and vice verse.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#228 - 2012-07-23 15:47:29 UTC
Shellac Brookdale wrote:
... But can we please have something like this for killmails too? Have EVEmon users auto upload killmails into a separate data relay network and enjoy all kinds of new killmail consuming apps making use of this? Making it easier for killboards to sync kms? That would be huge.

While this would certainly be useful, I am just stretched way too far to undertake it myself. This would need to be a separate entity from EMDR, but a huge chunk of the codebase could be re-used trivially. Basically the only thing that would need changing would be the gateway daemon. Instead of looking for market data, it'd need to use whatever format the uploads come over as.

I'd love to offer advice and pointers for anyone wishing to undertake this, so poke me if you do end up giving it a shot. The EMDR codebase and ZeroMQ are both relatively simple (and BSD-licensed, extremely permissive), and much of the tedious testing and messing with relays/announcers has been done for you (those could both be re-used without modification). An adaptation of EMDR to work as a new service could be done over a weekend of solid effort.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#229 - 2012-07-23 15:48:10 UTC
EMDR has added relays in Denmark and Holland, courtesy of Karbowiak:

http://www.eve-emdr.com/en/latest/access.html

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Qualla Amborante
Royal Amarr Institute
Amarr Empire
#230 - 2012-08-03 08:49:09 UTC
The Holland relay does not work, does not accept the connection. Gemany works fine.

Thnx.
Qualla Amborante
Royal Amarr Institute
Amarr Empire
#231 - 2012-08-03 10:44:59 UTC
Hello,

Can I tell the feed somehow that I do not want the historical data? If I get all data, and save it, I do also have this history, and save a load of bandwith, or am I mssing something?

Qualla
Kaladr
Viziam
Amarr Empire
#232 - 2012-08-04 02:47:25 UTC  |  Edited by: Kaladr
Desmont McCallock wrote:
I thought that you might ask. Killmail API is defected as hell. If you have given to a service (site or client) an API key to retrieve kill mails, you can't retrieve the same info (even with another API key) so to use to another service. Meaning that the killmail API call doesn't return persisting data.

This is also the reason I haven't added a kill mail monitor in EVEMon yet. Those that use any API key to feed a killboard won't be able to use EVEMon (or any other app/service that use the same API call) to monitor the kill mails and vice verse.


I'd love to process the killmail data in the future. However, in many ways, killmails are more of a state syncing problem instead of a pure stream. You can build a whole view of the market by watching the EMDR feed for awhile, but you can't build a whole "view" of killmails. In many ways RSS is a good concept, but it doesn't really scale.

(And to get on the topic of EMDR, EVE-Central is now relaying EMDR data, unofficially at emdr.eve-central.com:8050, and officially once the docs get updated).

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#233 - 2012-08-04 19:28:04 UTC
Sorry for the delay, Kaladr's west coast relay is now officially listed here:

http://www.eve-emdr.com/en/latest/access.html

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#234 - 2012-08-24 20:01:01 UTC
We're in need of an announcer machine. If you'd like to volunteer one and have a good chunk of bandwidth to spare, please see: https://groups.google.com/forum/?fromgroups=#!topic/eve-emdr/JXMr5iqZGbU

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Vaerah Vahrokha
Vahrokh Consulting
#235 - 2012-10-31 08:14:36 UTC
Hello,

I'd like to attach to EMDR and fetch the data but I have a problem: my hosting won't let me install daemons or perma-running processes that can sit listening to sockets and similar. I can only run a cron.

Is there any way I can deal with this restriction and still get data?

I noticed that the peeps at Element43.com seem to be hosted on GoDaddy yet they can process EMDR feeds, so there seems to be a way.

EvE Online portal | EvE markets tutorials
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#236 - 2012-11-01 16:49:34 UTC
Vaerah Vahrokha wrote:
Hello,

I'd like to attach to EMDR and fetch the data but I have a problem: my hosting won't let me install daemons or perma-running processes that can sit listening to sockets and similar. I can only run a cron.

Is there any way I can deal with this restriction and still get data?

I noticed that the peeps at Element43.com seem to be hosted on GoDaddy yet they can process EMDR feeds, so there seems to be a way.


There's not a great way to do this. Your only hope would be to do some pretty hacky stuff with a long (permanently) running cron entry, but GoDaddy would probably get irate with you.

Element43 isn't on GoDaddy.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Vaerah Vahrokha
Vahrokh Consulting
#237 - 2012-11-02 10:55:56 UTC  |  Edited by: Vaerah Vahrokha
Ilyk Halibut wrote:
Vaerah Vahrokha wrote:
Hello,

I'd like to attach to EMDR and fetch the data but I have a problem: my hosting won't let me install daemons or perma-running processes that can sit listening to sockets and similar. I can only run a cron.

Is there any way I can deal with this restriction and still get data?

I noticed that the peeps at Element43.com seem to be hosted on GoDaddy yet they can process EMDR feeds, so there seems to be a way.


There's not a great way to do this. Your only hope would be to do some pretty hacky stuff with a long (permanently) running cron entry, but GoDaddy would probably get irate with you.

Element43 isn't on GoDaddy.


Thank you.

I am already invested in a certain web hosting company (no daemon support sadly) for other websites so all the EvE software and websites - I make all for free - are going to be a pure additional expense to me.

Therefore I'd love if you or some other experienced programmer in there could suggest me a cheap hosting service that lets me install such a permanently running application needed to listen to ZeroMQ messages.
I have checked Webfaction and Dreamhost but both are expensive for totally free EvE projects. Plus they really push into multi-year subscription purchases and I can't commit to such a budget.
All I needed to do is to listen for EMDR messages 24/7 and slap them in a database, I don't even need a web frontend (no I can't do it with a computer at home).

EvE Online portal | EvE markets tutorials
Muscaat
EVE Markets
#238 - 2012-11-02 12:36:43 UTC
Vaerah Vahrokha wrote:
I'd love if you or some other experienced programmer in there could suggest me a cheap hosting service that lets me install such a permanently running application needed to listen to ZeroMQ messages.
I have checked Webfaction and Dreamhost but both are expensive for totally free EvE projects. Plus they really push into multi-year subscription purchases and I can't commit to such a budget.
All I needed to do is to listen for EMDR messages 24/7 and slap them in a database, I don't even need a web frontend (no I can't do it with a computer at home).

I'm in a similar position to you, Vaerah - my host doesn't want persistent processes but I'd like to collect EMDR data. I went for a cheap VPS - there are some really cheap packages around. http://www.lowendbox.com/ is a good place to start looking. We're talking down to about US$3/mo in some cases. Might be worth a try.
Desmont McCallock
#239 - 2012-11-02 13:48:39 UTC
Muscaat wrote:
I'm in a similar position to you, Vaerah - my host doesn't want persistent processes but I'd like to collect EMDR data. I went for a cheap VPS - there are some really cheap packages around. http://www.lowendbox.com/ is a good place to start looking. We're talking down to about US$3/mo in some cases. Might be worth a try.
Good to know you are still around. Have you noticed that your stats page is broken?
Vaerah Vahrokha
Vahrokh Consulting
#240 - 2012-11-02 14:03:47 UTC
Muscaat wrote:
Vaerah Vahrokha wrote:
I'd love if you or some other experienced programmer in there could suggest me a cheap hosting service that lets me install such a permanently running application needed to listen to ZeroMQ messages.
I have checked Webfaction and Dreamhost but both are expensive for totally free EvE projects. Plus they really push into multi-year subscription purchases and I can't commit to such a budget.
All I needed to do is to listen for EMDR messages 24/7 and slap them in a database, I don't even need a web frontend (no I can't do it with a computer at home).

I'm in a similar position to you, Vaerah - my host doesn't want persistent processes but I'd like to collect EMDR data. I went for a cheap VPS - there are some really cheap packages around. http://www.lowendbox.com/ is a good place to start looking. We're talking down to about US$3/mo in some cases. Might be worth a try.


Thank you, much appreciated! I can shell $3 for something I won't ever get a nickel from, but $12 was a bit too steep (unless going for a lifelong pre-paid plan then it drops to $8).

EvE Online portal | EvE markets tutorials