These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Real-time market data service: EVE Market Data Relay
Author
Previous Topic Next Topic
Desmont McCallock
#81 - 2012-04-22 11:57:47 UTC
Snarf Aldes wrote:
Question.. Why are you reformatting all the incoming data?
I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID.
Here is an example split_json.txt
The top is what i am sending, the bottom is what is returned through EMDR.

I really need it to be in the top format, i cannot use the bottom format What?
"uploadKeys":[{"name":"Eve Addicts","key":"relay"}],
should be
"uploadKeys":[{"name":"EVE Market Data Relay","key":"EMDR"}],
as uploadKeys state where you are uploading.
Snarf Aldes
University of Caille
Gallente Federation
#82 - 2012-04-22 12:24:18 UTC
Desmont McCallock wrote:
Snarf Aldes wrote:
Question.. Why are you reformatting all the incoming data?
I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID.
Here is an example split_json.txt
The top is what i am sending, the bottom is what is returned through EMDR.

I really need it to be in the top format, i cannot use the bottom format What?
"uploadKeys":[{"name":"Eve Addicts","key":"relay"}],
should be
"uploadKeys":[{"name":"EVE Market Data Relay","key":"EMDR"}],
as uploadKeys state where you are uploading.

We can agree to use it that way, that would solve the duplicate insertion problem :)
But.. i don't think that is the intended purpose of uploadKeys.
What the actual purpose of uploadKeys is is unclear to me.

Creator of Eve Addicts
Desmont McCallock
#83 - 2012-04-22 13:10:12 UTC  |  Edited by: Desmont McCallock
From spec docs:
Quote:
uploadKeys; array; An array of objects containing per endpoint upload keys
Endpoint is the entity to where you are uploading.

For example, once EVEMon Market Unified Uploader go live I will have to obtain an upload key from you so you can identify it and give credits to it.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#84 - 2012-04-22 18:50:48 UTC
Snarf Aldes wrote:
Question.. Why are you reformatting all the incoming data?
I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID.
Here is an example split_json.txt
The top is what i am sending, the bottom is what is returned through EMDR.

I really need it to be in the top format, i cannot use the bottom format What?

Sorry about this, I hunted it down and corrected this just now. It won't be a problem anymore.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#85 - 2012-04-22 18:54:14 UTC
Snarf Aldes wrote:

We can agree to use it that way, that would solve the duplicate insertion problem :)
But.. i don't think that is the intended purpose of uploadKeys.
What the actual purpose of uploadKeys is is unclear to me.

Since EMDR has no notion of an upload key, I just made that up from my scraper processes. I'm not actually modifying data, there just is no such thing as an upload key when I'm pulling the data from EC/EMD's feeds/API, and figured it might be useful to someone to see it in that format.

We probably do need to keep experimenting with solutions to the upload loop problem. I don't think the discussion is closed yet, and I'm willing to go outside the spec if need be.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Wind Jammer
Molden Heath Software Company
#86 - 2012-04-22 22:07:49 UTC
Nice work Ilyk. Your site looks good, and the architecture seems well thought through.

It's probably not ideal for my needs because I need an aggregator. I don't want to have to calculate my own averages or identify or filter out manipulation attempts. And I'm happy collecting new data once a day. But good luck all the same. Smile


http://eveproductionmixer.appspot.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#87 - 2012-04-22 23:45:18 UTC
Wind Jammer wrote:
Nice work Ilyk. Your site looks good, and the architecture seems well thought through.

It's probably not ideal for my needs because I need an aggregator. I don't want to have to calculate my own averages or identify or filter out manipulation attempts. And I'm happy collecting new data once a day. But good luck all the same. Smile

Thanks! I guess the good thing is that you'll still benefit from it indirectly, in that wherever you pull your averages from will have better data.

Best of luck with your project.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Snarf Aldes
University of Caille
Gallente Federation
#88 - 2012-04-23 10:19:47 UTC
Ilyk Halibut wrote:
Snarf Aldes wrote:
Question.. Why are you reformatting all the incoming data?
I just noticed that the JSON i'm uploading (single rowset per typeID) is transformed into the split version with 2 rowsets for the same typeID.
Here is an example split_json.txt
The top is what i am sending, the bottom is what is returned through EMDR.

I really need it to be in the top format, i cannot use the bottom format What?

Sorry about this, I hunted it down and corrected this just now. It won't be a problem anymore.

Cool, I switched the EMDR stream on again.
I am seeing duplicates for every entry now, but that could be because the same data is originating from different sources.

Creator of Eve Addicts
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#89 - 2012-04-23 14:10:46 UTC
Snarf Aldes wrote:

Cool, I switched the EMDR stream on again.
I am seeing duplicates for every entry now, but that could be because the same data is originating from different sources.

Yeah, you should see up to three duplicates, since the Unified Uploader uploads to EVE Central, EVE Marketeer, and EVE Marketdata. We just started getting redirected downloads from EVE Marketeer, so this is more noticeable, since we cover all three.

This bothered me initially, but it does allow consumers to cross-check data to make sure some dufus like Milla isn't uploading bad data.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#90 - 2012-04-24 04:21:09 UTC
Pretty tame day today:

  • Made the Uploading documentation more specific, in that Contribtastic for Mac is the only client I know that supports Unified Uploader right now. There is upcoming support for others, do let me know if I've missed any that currently support it.
  • Fixed a bug with uploadKeys being a dict instead of a list for EMK uploads.
  • Improved validation to prevent issues with uploadKeys.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#91 - 2012-04-25 04:27:04 UTC
If all continues to run quietly, I'll probably aim to tag "1.0" this weekend. I'd love some more eyes on the feed to make extra sure that everything is in order leading up to then. After that point, I'll be looking for people willing to volunteer additional gateways and/or relays.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#92 - 2012-04-25 08:42:15 UTC
I build a script into the EMDR client to detect any bad formatted UF message.
Until now every message comes in correct format.
Desmont McCallock
#93 - 2012-04-25 15:33:08 UTC
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#94 - 2012-04-29 04:09:34 UTC
I'm moving houses this weekend/week, but snuck in an update to the EVE Central SMTP feeder to address occasional funkages, after I noticed we had lower inbound order data from them.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#95 - 2012-05-01 18:56:22 UTC
The real-time map appears to be borked atm. We're aware of this, and I've harassed the author to fix :)

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#96 - 2012-05-02 05:37:35 UTC
Tonight's updates:

  • The real-time map has been un-borked: http://map.eve-emdr.com
  • Worked with the EVE Central maintainer to correct their broken SMTP feed. We're getting some data, but it is currently sending out badly formed data (that we are filtering) at intervals. Talked to Yann about just using a direct ZeroMQ feed. Which leads to the next big development...
  • I've got a prototype ZeroMQ gateway (IE: You can send orders to us over ZeroMQ instead of HTTP) running. Please PM me if you're interested. I'll have documentation soon.
  • Further refinements to the uploader-visible error feedback.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#97 - 2012-05-12 03:24:34 UTC
Recent changes:

  • Added an experimental ZMQ gateway for sites that would like to relay market data more efficiently. Please PM me if you're interested. It's still preliminary, but I'd love to have help testing.
  • Hopefully improved handling of gzipped uploads.
  • Supporting the silly 'data' POST form-encoded requests.
  • Lots of creature comforts with eventual machine volunteers in mind. All of the startup scripts are much improved.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com
Desmont McCallock
#98 - 2012-05-12 15:47:54 UTC
Code changes for post formed and compressed uploads are incorrect. Get in touch and I'll walk you through the fixes.
Barl Rathbone
Corel Industries
#99 - 2012-05-12 16:18:39 UTC  |  Edited by: Barl Rathbone
Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.

My rackspace stuff is all sourced out of the Chicago datacenter. Or if needed I could bring up a Linode, looks like that's what you're using currently.
Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#100 - 2012-05-14 04:07:55 UTC
Barl Rathbone wrote:
Ilyk, if you still want/need another relay I can set up a new node for you on my rackspacecloud account. I assume it doesn't have to be a large one.

My rackspace stuff is all sourced out of the Chicago datacenter. Or if needed I could bring up a Linode, looks like that's what you're using currently.

I'll definitely be in need of relays in the near future. Been dealing with moving into a new house unexpectedly, which slowed things down for a while, but I'm back in action now. Aiming to get one or two more issues worked through this week, then do a fial documentation push, then look to get some relays set up.

Rackspace or Linode are both just fine. It just can't be Atlanta+Linode, since that wouldn't give us any useful additional redundancy. Data usage from Atlanta to another Linode location may be free, though.

I'll be sending the call out for relays on the forums, and on the mailing list: https://groups.google.com/forum/#!forum/eve-emdr

Keep an eye on whichever one works best for you. The mailing list is super, super low volume, as I've mostly been doing things here. That'll probably be more for important announcements once we're up and running.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com