These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Why are we not combining market websites, player devs?

Author
Callean Drevus
Perkone
Caldari State
#1 - 2012-06-03 15:42:28 UTC
Now that there is the unified upload format, now that we have an uploader in EVEMon, and now that we have EMDR to stream it all, why are we all divided when it comes to building market websites?

I realize that everone has their own way of doing things, their own preferred technologies, and their own ideas on how to build websites, but since we overcame all those things with creating the unified upload format, I was wondering whether we might be able to do it with creating a market website.

We have always been able (for some websites) to contribute to an already existing website, but I have found that the difficulty with doing so is that you feel it is someone else's baby already. They might accept your code, or they might reject it for any number of (valid or invalid) reasons.

I know I have things I'd rather not see in EMK, but I've also seen lots of features on other websites, that I haven't been able to produce myself, or that I think would be awesome, if given a decent layout. I also have lots of ideas on how I would redo a market website if I would start from scratch again, and there are probably lots of developers out there whom would be able to give me very good suggestions on how to do it even better.

Therefore, I pose to you the question: do you think we would be able to build a common market website together, pooling development resources, and quite possibly building the best site yet? If not, why? If yes, how do you propose we go about it?

This is just a question to the community, whether anything comes of it is not an issue yet ;)

Developer/Creator of EVE Marketeer

Dragonaire
Here there be Dragons
#2 - 2012-06-07 14:42:42 UTC
I think there's a couple problems that would have to be overcame to do what you're talking about.

First everybody seems to start their project because they think they can do it better than everyone else Blink
Second they all also seem to do it because they wanted to learn how to do it for themselves.

Sometimes the order is reversed and there are many other reasons as well but those two seem to be very common reasons for most Eve third party developers and both of them can cause problems when trying to work as a group but that's not to say it can't happen just that it'll take a lot more work and probably one or two people to direct it and make sure everyone plays nice together Blink

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#3 - 2012-06-07 22:15:51 UTC
I think you may have hit the nail on the head with the differing approaches being the primary motivator. For example, I absolutely loathe PHP (which is my completely unreasonable, flawed, subjective opinion), and my free time has to be spent on things that i enjoy (not PHP). Some want to use Ruby on Rails, others like Scala, and my own tastes gravitate towards Python web frameworks. That's a lot of subjective preferences to reconcile!

As you have noted, working on a market site is a very time-consuming process with a very low probability of long-term survival. There are just too many moving parts for one or two people to pay for, develop, maintain, and continue to run on top of real life, relationships, and other things. Fortunately, we can ease the burden for everyone and better the whole.

One of the huge motivations behind EMDR was to take part of that market data site "stack" and "outsource" it to the point where market site developers don't need to deal with uploader politics and gathering their own swarms of players to submit market data. It doesn't matter what language EMDR is written in, since we have language-agnostic communication between producers and consumers. EMDR is designed to not be a heavy financial burden on me or any of the volunteers, hopefully ensuring a long life. I think these combination of attributes are the potential way to approach sustainable community development. Language-agnosticism, a shared and lesser financial burden for volunteers, and extreme specialization (it only does a few things). And of course, such an effort has to be 100% open source for it to survive long-term.

I think the next steps towards your more community-driven ideal would be to "outsource" additional components that typical market sites re-invent from site to site. You probably know these better than I do, but it seems like a lot of EMDR users aren't sure of the best way to store and analyze market data. Perhaps that would be a good barrier to overcome together as a community. Maybe we write a super-smart and paranoid consumer so others don't have to. Maybe we serve the data in a highly performant, reliable manner (more so than the current multi-purpose market sites can). If we build re-usable, language-agnostic components that developers can plug in, we achieve the community-maintained ideal (since it's mutually beneficial for all of us to improve on them), and we empower everyone.

Small, simple pieces are the key, IMHO. Build a small, simple, component, make it reachable/usable from almost any language, build a community around it, and repeat. Keep doing this and I think we move towards convergence.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#4 - 2012-06-07 22:28:51 UTC
To provide some more food for thought, here's a post on my blog with some of my doodlings that eventually led to the creation of EMDR: http://gc-taylor.com/blog/2012/3/11/more-efficient-market-web-apis-eve-online/

It was no more than a brain exercise for the most scalable way I could thing to make market data available. The best attribute was that the service this post describes is "infinitely" scalable, something like 99.9999% or better uptime, and can optionally sit behind a CDN (CloudFront?) to distribute the data to sites even faster. The downside is that it is a financial burden on a single person, which is why I never acted on it. The alternative would be to use DevPay and have people pay for the bandwidth they use, but people are cheap, and there are already other freely available APIs to use (even with their lesser performance and uptime).

This may or may not be useful at all, but it's useful to look back on past musings. Perhaps EMDR has helped to make this kind of effort less appealing these days, since it's so easy to collect and record your own data now. The analysis is probably the hardest part now.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#5 - 2012-06-09 04:10:15 UTC
Another potentially useful idea would be a community-developed EMDR consumer or market data accepting gateway that did de-duping and validation/cross-checking of incoming data. It could spit JSON back out over ZMQ or write directly to a DB for later aggregation and statistical analysis.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Callean Drevus
Perkone
Caldari State
#6 - 2012-06-15 07:05:16 UTC
Hehe, I think most points raised are indeed valid. That said, just making small scalable components in whatever language you feel is best, is a nice compromise (I like the part about working towards convergence, which may well be true).

Developer/Creator of EVE Marketeer

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#7 - 2012-06-15 13:51:55 UTC
Callean Drevus wrote:
Hehe, I think most points raised are indeed valid. That said, just making small scalable components in whatever language you feel is best, is a nice compromise (I like the part about working towards convergence, which may well be true).

Along that same line, I just released a set of EVE market data structures and serializers (to/from Unified format) here: https://github.com/gtaylor/EVE-Market-Data-Structures

That could be re-used for uploaders, HTTP uploader accepting gateways, things like EMDR, and market sites (consumers). Pretty much no documentation but the unit tests, but hey, release early, release often :)

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com

Ilyk Halibut
Deep Core Mining Inc.
Caldari State
#8 - 2012-06-22 18:39:04 UTC
Another random idea. A market data API module that uses this instead of HTTP:

http://zerorpc.dotcloud.com/

Applying what we've learned from EMDR, this is potentially a lower overhead, lower latency alternative to making your market data available to developers.

EVE Market Data Relay - A real-time feed of EVE Market data http://www.eve-emdr.com