These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EMDR related question, to implement a workflow
Author
Previous Topic Next Topic
Vaerah Vahrokha
Vahrokh Consulting
#1 - 2013-08-20 12:49:42 UTC
Hello,

I am here to ask the knowledgeable guys lurking in this forum for a suggestion.

I need to attach to an EMDR datafeed.

I am interested only in the data for 1 market in The Forge. So far nothing special.

I need to access the exact price of the lowest current sell order that is currently active for that market.

This process of extracting the current lowest sell order price needs to be repeated at every new order. Every hour the stats are reset.

Is the logic just about checking something like:

if (newly_arrived_order_active && newly_arrived_order_type_is_sell) {
add_to_stats(newly_arrived_order_price);
}

Because somehow I feel EvE's orders system is more complex than this.

Thanks in advance.

EvE Online portal | EvE markets tutorials
Ydnari
Estrale Frontiers
#2 - 2013-08-20 18:33:18 UTC
You'll also want to check (at least) the generatedAt attribute of the EDMR message as messages may arrive way out of order, not least because someone's instance of EVEMon may upload an old cache file because it didn't happen to be running at the time it was generated.

Other than that, pretty much that simple.

You can't request the EMDR datafeed is filtered at source so you get the entire firehose and you have to filter it your side, but that's probably not a problem.

--
Vaerah Vahrokha
Vahrokh Consulting
#3 - 2013-08-20 20:18:59 UTC
Ydnari wrote:
You'll also want to check (at least) the generatedAt attribute of the EDMR message as messages may arrive way out of order, not least because someone's instance of EVEMon may upload an old cache file because it didn't happen to be running at the time it was generated.

Other than that, pretty much that simple.

You can't request the EMDR datafeed is filtered at source so you get the entire firehose and you have to filter it your side, but that's probably not a problem.


Thank you so much!

To recap, I am going to accept only orders with active flag + marked as sell order plus date >= the date of my last received order, which in turn must fit within the 1 hour "chunks" I am splitting the data into.
As for the "firehose", I'll deal with it.

EvE Online portal | EvE markets tutorials
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#4 - 2013-08-20 21:06:59 UTC
If you can read perl

https://github.com/fuzzysteve/EMDR-Consumer/blob/master/sell.pl

May be of interest. It takes all sell orders, tosses them if they're older than the currently stored date, then aggregates the data before storing it. It pays attention to the region, but only for renaming one (for historical (and laziness) reasons). Simple enough just to skip onto the next order if it wasn't the region you wanted.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
scrambled
Doomheim
#5 - 2013-08-20 23:49:16 UTC
Steve Ronuken wrote:
If you can read perl

https://github.com/fuzzysteve/EMDR-Consumer/blob/master/sell.pl

May be of interest. It takes all sell orders, tosses them if they're older than the currently stored date, then aggregates the data before storing it. It pays attention to the region, but only for renaming one (for historical (and laziness) reasons). Simple enough just to skip onto the next order if it wasn't the region you wanted.


Any chance you can twiddle EMDR::Consumer to be able to be used with AnyEvent or Mojo::IOLoop? Or rather, mind if I do and submit a pull request?
Vaerah Vahrokha
Vahrokh Consulting
#6 - 2013-08-20 23:51:22 UTC
Thank you too!
I am going to vote for you if you really go for CSM.

As for the PERL, this is easy to read.
I still recall when I had to setup some PERL payment gateways in the 90s, at the time PERL = a program 10 lines long, 8 of which were 1 regular expression Lol

EvE Online portal | EvE markets tutorials
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#7 - 2013-08-21 02:19:43 UTC
Vaerah Vahrokha wrote:
Thank you too!
I am going to vote for you if you really go for CSM.

As for the PERL, this is easy to read.
I still recall when I had to setup some PERL payment gateways in the 90s, at the time PERL = a program 10 lines long, 8 of which were 1 regular expression Lol

I try not to obfuscate my code Smile makes it a pain to adjust later.

scrambled wrote:
Steve Ronuken wrote:
If you can read perl

https://github.com/fuzzysteve/EMDR-Consumer/blob/master/sell.pl

May be of interest. It takes all sell orders, tosses them if they're older than the currently stored date, then aggregates the data before storing it. It pays attention to the region, but only for renaming one (for historical (and laziness) reasons). Simple enough just to skip onto the next order if it wasn't the region you wanted.


Any chance you can twiddle EMDR::Consumer to be able to be used with AnyEvent or Mojo::IOLoop? Or rather, mind if I do and submit a pull request?


Uh, feel free? Smile

I've not looked into any of the event frameworks (as I've not had a reason to, in the past). This has always worked well enough for my goals.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
scrambled
Doomheim
#8 - 2013-08-21 12:10:06 UTC
Steve Ronuken wrote:
Vaerah Vahrokha wrote:
Thank you too!
I am going to vote for you if you really go for CSM.

As for the PERL, this is easy to read.
I still recall when I had to setup some PERL payment gateways in the 90s, at the time PERL = a program 10 lines long, 8 of which were 1 regular expression Lol

I try not to obfuscate my code Smile makes it a pain to adjust later.

scrambled wrote:
Steve Ronuken wrote:
If you can read perl

https://github.com/fuzzysteve/EMDR-Consumer/blob/master/sell.pl

May be of interest. It takes all sell orders, tosses them if they're older than the currently stored date, then aggregates the data before storing it. It pays attention to the region, but only for renaming one (for historical (and laziness) reasons). Simple enough just to skip onto the next order if it wasn't the region you wanted.


Any chance you can twiddle EMDR::Consumer to be able to be used with AnyEvent or Mojo::IOLoop? Or rather, mind if I do and submit a pull request?


Uh, feel free? Smile

I've not looked into any of the event frameworks (as I've not had a reason to, in the past). This has always worked well enough for my goals.



Yeah, just that I have an existing app that uses Mojolicious that would benefit from having EMDR access - and to keep it fast, non-blocking is the way to go. I'll give it a shot, was working on it yesterday but code that works when you run it straight from a script suddenly refuses to work right when you move it into a module, ZMQ starts throwing errors left right and center.

I'll thwap it some more, will throw a pull request up when it works :)
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#9 - 2013-08-21 12:13:19 UTC
You might want to keep it separate, and only access the data that you're getting from the feed. For it to be useful data, you'll have to be storing it in something persistent anyway.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Arch Stanton's Neighbour
Forceful Resource Acquisition Inc
#10 - 2013-08-21 14:39:50 UTC  |  Edited by: Arch Stanton's Neighbour
I have a custom running program that feeds off the EMDR so a couple pointers from me:

1 - you can't just compare the lastest order's price to a minimum price because you have no way of knowing if the order your minimum price came from is still active
2 - therefore you need to know which orders are active. this is actually impossible with EMDR because it doesn't tell you when an order expires
3 - therefore you need to keep a cache of orders that you have recently seen on the data feed (i found 2 hours is the window that works best) and at any given time your mininum sell price will be the mininum from that cache of orders less than two hours old.

Or just get it from eve-central's api which does the legwork and gives you basically the same result....

HTH.
Vaerah Vahrokha
Vahrokh Consulting
#11 - 2013-08-21 18:30:06 UTC
Arch Stanton's Neighbour wrote:
I have a custom running program that feeds off the EMDR so a couple pointers from me:

1 - you can't just compare the lastest order's price to a minimum price because you have no way of knowing if the order your minimum price came from is still active
2 - therefore you need to know which orders are active. this is actually impossible with EMDR because it doesn't tell you when an order expires
3 - therefore you need to keep a cache of orders that you have recently seen on the data feed (i found 2 hours is the window that works best) and at any given time your mininum sell price will be the mininum from that cache of orders less than two hours old.

Or just get it from eve-central's api which does the legwork and gives you basically the same result....

HTH.


Thank you for your kind suggestion.

I think I am spared off this scenario, as I only have to record four instant values per given timeframe, it's not important if they become inactive after the sample.

EvE Online portal | EvE markets tutorials
Desmont McCallock
#12 - 2013-08-21 18:40:33 UTC  |  Edited by: Desmont McCallock
Arch Stanton's Neighbour wrote:
I have a custom running program that feeds off the EMDR so a couple pointers from me:

1 - you can't just compare the lastest order's price to a minimum price because you have no way of knowing if the order your minimum price came from is still active
2 - therefore you need to know which orders are active. this is actually impossible with EMDR because it doesn't tell you when an order expires
3 - therefore you need to keep a cache of orders that you have recently seen on the data feed (i found 2 hours is the window that works best) and at any given time your mininum sell price will be the mininum from that cache of orders less than two hours old.

Or just get it from eve-central's api which does the legwork and gives you basically the same result....

HTH.

Regarding point 2, the expression 'expires' isn't correct. You can know when the order expires as IssueDate and Duration is included in the order's data, but certainly you can not know when that order gets fulfilled. The only way is to compare the region orders between two or more feeds in time.
Arch Stanton's Neighbour
Forceful Resource Acquisition Inc
#13 - 2013-08-21 18:53:04 UTC
Yes, that's what I meant, there is no way to know when it is fulfilled or cancelled. Thanks for the addendum.