These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
123Next page
 

A challenge with no prize!

First post First post
Author
CCP FoxFour
C C P
C C P Alliance
#1 - 2015-02-04 16:01:23 UTC  |  Edited by: CCP FoxFour
Why... because why not? No really because I thought it would be fun to have something a bit more specific to work on. So here is the challenge: Build a static web application for browsing the EVE market using CREST.


  • The application should only contact CREST, aside from loading libraries such as jQuery or Angular, and should not rely on any other services.
  • Should show all items on the market.
  • Should allow a user to see the items buy and sell information.
  • Should be properly RESTful and follow links, sent proper accept headers, etc.


Things you will probably need:


My current idea on how I will try doing this:

  • Load all the regions
  • Load the market groups
  • Build the tree structure out of that
  • When a group is expanded check for types in that group and load them
  • When a type is clicked load the market data for that type


Or something like that anyways is what I plan on trying to do. I am personally going to try and get something done this weekend. If anyone else decides to do something post your results here. :)

If you go ahead and complete something, and come to FanFest, look me up while out one night and I will see about getting you a drink.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Daimian Mercer
Deep Core Mining Inc.
Caldari State
#2 - 2015-02-04 16:15:23 UTC
I shall heed the call!

Creator of Tripwire mapping tool - EVE-O thread

Twitter | daimian.mercer@gmail.com

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#3 - 2015-02-04 16:16:44 UTC
Already started working something up Smile


(Implicit flow for oauth as a tip. No need for the secret)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

SJ Astralana
Syncore
#4 - 2015-02-04 20:04:40 UTC  |  Edited by: SJ Astralana
The EVE client has the benefit of SQL calls, but we as devs have to slog through each item It takes a couple minutes using parallel processing to simply load T1 and related materials for a single region.

I'll buy YOU a drink if you would please allow requesting more than one item per call. Eve-MarketData and Eve-Central both are WAY faster than CREST.

Hyperdrive your production business: Eve Production Manager

Ortho Loess
Gemini Talon
Curatores Veritatis Alliance
#5 - 2015-02-04 20:04:52 UTC
Very tempted to join in.

No idea what you mean here:
Steve Ronuken wrote:
(Implicit flow for oauth as a tip. No need for the secret)

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#6 - 2015-02-04 20:33:15 UTC
Ortho Loess wrote:
Very tempted to join in.

No idea what you mean here:
Steve Ronuken wrote:
(Implicit flow for oauth as a tip. No need for the secret)




Once I have something up and running, I'll share it, and you can see Blink

So far, so good.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

CCP FoxFour
C C P
C C P Alliance
#7 - 2015-02-04 20:33:31 UTC
SJ Astralana wrote:
The EVE client has the benefit of SQL calls, but we as devs have to slog through each item It takes a couple minutes using parallel processing to simply load T1 and related materials for a single region.

I'll buy YOU a drink if you would please allow requesting more than one item per call. Eve-MarketData and Eve-Central both are WAY faster than CREST.


You're looking at this all wrong. You're looking at it as if building a service that needs all the information at once to do some kind of processing of that information. Please keep in mind CREST was built for allowing more clients access to TQ, not for building other services. For example DUST uses CREST and it doesn't take minutes to load the market. Your instinct to go straight to us having the benefit of SQL is also indicating how wrong you are looking at this. Instead of trying to prime all the information at once, as you are suggesting, start with two calls. One for the list of regions and one for the list of market groups. Then build the market tree from that and a selection for region. When someone clicks on a market group load the list of types for that group, again one call. Continue this until they click on a type and at that point load the buy and sell orders for that type for the region selected.

That is basically how the EVE client works as well. We only ever send to the client one types buy or sell orders at a time. The EVE and DUST client both do this. The only advantage the EVE client has is having the market groups store locally and knowing that you only get to see the region you are in.

This is exactly what CREST was built for and it performs this job fantastically. It just requires thinking about building a different kind of client.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

SJ Astralana
Syncore
#8 - 2015-02-04 21:18:13 UTC
CCP FoxFour wrote:
SJ Astralana wrote:
The EVE client has the benefit of SQL calls, but we as devs have to slog through each item It takes a couple minutes using parallel processing to simply load T1 and related materials for a single region.

I'll buy YOU a drink if you would please allow requesting more than one item per call. Eve-MarketData and Eve-Central both are WAY faster than CREST.


You're looking at this all wrong.


Actually, you're looking at this all wrong. Robustness means a service can be used in unexpected ways and still excel. You treat the scenario you designed as THE scenario, where it actually is A scenario. The third party services see that and hence are more robust.

CREST is poor at providing the information required for the following report for all buildable T1 items in Amarr: This is probably the most compelling scenario in production: what to build. And nobody does it worse than CREST presently.

Market Research

Item Isk Per Hour Margin Markup Volume Inventory Profit Factor
Caldari Control Tower 4,436,663 100,833,259 35.7 5.5 5 110,244,400.
Amarr Control Tower 5,003,735 113,721,249 40.3 7.4 11 76,503,400.
Amarr Control Tower Medium 4,236,950 64,196,211 45.9 6.4 11 37,350,500.
Equipment Assembly Array 6,209,704 18,817,286 51.6 5.1 3 31,989,400.
Minmatar Control Tower 3,336,663 75,833,258 26.9 2.7 8 25,909,700.
Arkonor Mining Crystal I 5,704,344 432,147 177.9 54.7 1 23,624,000.
X-Large Ship Maintenance Array 46,919,014 142,178,832 21.5 0.9 7 17,603,100.
Caldari Control Tower Medium 1,926,950 29,196,211 20.9 4.0 7 16,683,500.
Capital Hybrid Metastasis Adjuster I 11,451,766 17,351,160 1,052.3 1.3 2 11,567,400.
Gallente Control Tower 1,088,233 24,732,560 8.8 1.8 4 11,129,700.

Hyperdrive your production business: Eve Production Manager

Schmell
Russian Thunder Squad
Pandemic Legion
#9 - 2015-02-04 21:39:25 UTC  |  Edited by: Schmell
Quote:
One for the list of regions and one for the list of market groups. Then build the market tree from that and a selection for region. When someone clicks on a market group load the list of types for that group, again one call. Continue this until they click on a type and at that point load the buy and sell orders for that type for the region selected.


That's all cool and stuff, but some options for bulk downloading data wouldn't hurt tbh.

Right now i am building simple app for alliance tournament replay analysis. Browsing tournaments and teams is done exactly as you suggest, but getting replays (one frame per request) certainly is not as fast, as i initially wanted it to be
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#10 - 2015-02-04 21:57:51 UTC
https://www.fuzzwork.co.uk/market/viewer2/

\o/

(for some reason my firefox doesn't like it. But it runs fine in Chrome)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Ortho Loess
Gemini Talon
Curatores Veritatis Alliance
#11 - 2015-02-04 23:22:40 UTC
Bloody hell steve, that was quick!

I get not found or denied when selecting items. I'll just assume that you've not implemented that part yet though. Smile
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#12 - 2015-02-05 00:07:49 UTC  |  Edited by: Steve Ronuken
Ortho Loess wrote:
Bloody hell steve, that was quick!

I get not found or denied when selecting items. I'll just assume that you've not implemented that part yet though. Smile



Really?

It's working for me What?

What items are you trying, and which browser are you using? And which region?



(make sure you fully reload everything. Just to make sure everything is up to date)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Ortho Loess
Gemini Talon
Curatores Veritatis Alliance
#13 - 2015-02-05 00:23:43 UTC
Steve Ronuken wrote:
What items are you trying, and which browser are you using? And which region?

(make sure you fully reload everything. Just to make sure everything is up to date)


I've refreshed everything. Still getting a mix of not found and forbidden (Sometimes one and then the other, on the same item.)

I tried The Forge, Domain and Providence. Tried Apoc and geddon in domain (that's the most specific combination I remember)

Google Chrome


I get the lists of market groups, which correctly expand, also get the items and their images.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#14 - 2015-02-05 00:36:34 UTC
That's weird

Just worked for me What?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Dreae
Deep Core Mining Inc.
Caldari State
#15 - 2015-02-05 00:49:29 UTC
Getting the same problem with Steve's implementation. A quick glance in the console reveals the JavaScript is requesting.
https://crest-tq.eveonline.com/market/10000069/orders/sell/?type=undefined

But it only occurs when clicking on the image for the item, as it seems openItem is getting the img element instead of the intended li, clicking on the text for the item works as intended. Adding pointer-events: none to the images should fix it for browsers that support it, or adding data-cresthref to the images.

I was planning on developing a JavaScript library for the API, and this sounds like a good excuse to do it P I'll try to get something worked up this weekend.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#16 - 2015-02-05 00:59:20 UTC
Ahhhhhh...

That explains it.

Updated now to pull if you click on the picture too Smile The image inherits the click handler from the li it's on. But as Dreae noted, it's looking for the cresthref data attribute on the clicked thing. That's now added to the img too.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Ortho Loess
Gemini Talon
Curatores Veritatis Alliance
#17 - 2015-02-05 01:01:46 UTC
Yup, works clicking on the text :) (Could have sworn I tried that, foolish me!)
Dragonaire
Here there be Dragons
#18 - 2015-02-05 04:53:34 UTC
SJ Astralana - Actually FoxFour is correct I think here you're just going at the problem wrong. When you're going to built something you don't look for everything on the market that can be built but what things you have BPO/BPCs for first. That's step one then from that you might look at a group of things to build like your example a control tower. At that point you probably want to know about the pricing info and can grab it from CREST. You may also at this point want to know the cost on the materials etc so using the SDE you'll built a list of items with market groups etc just like the client does and can get the current prices on everything and see if you think there's a profit and setup buy order and or/ start the job etc as needed. Note that at no point did you need every items available on the market and I'll guess that contacting CREST for the half dozen things you needed could all be done in less than a second. It'll take you long to decide what to build than to grab the actual info that is needed done this way.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

SJ Astralana
Syncore
#19 - 2015-02-05 06:47:38 UTC
Dragonaire wrote:
SJ Astralana - Actually FoxFour is correct I think here you're just going at the problem wrong. When you're going to built something you don't look for everything on the market that can be built but what things you have BPO/BPCs for first.


That's quite naive, thanks for playing. Workflow: Run market research report. Buy BPO if key indicators are in line. Research it. Profit. Yes there's a delay, but what BPO to buy follows from what's profitable. I just threw a new BPO in the oven today, joining 8 elder brothers. Of the ten best profit items available today (and I mean real profit, not some silly IPH comparison), six are online meaning their profitability has persisted over multiple months, two are in the oven because they match the profile of long-term profit, and the other two are transients.

EPM does do imports from CREST for the existing portfolio for a different report, so yeah I actually know how to limit a list of requests to only what's needed.

Hyperdrive your production business: Eve Production Manager

Kali Izia
GoomWaffe
#20 - 2015-02-05 07:27:55 UTC
Steve Ronuken wrote:
Already started working something up Smile


(Implicit flow for oauth as a tip. No need for the secret)

Implicit is great if you're going fully client-side like your app is. But if you have the option of using a server, in my opinion the regular flow is definitely preferable.

For something like a market browser where you're clicking around a lot and might have it open in the background while you're doing other stuff in-game or whatever, you'd probably hit the 20 minute expiry fairly easily.
123Next page