EVE Technology Lab

Now that there is the unified upload format, now that we have an uploader in EVEMon, and now that we have EMDR to stream it all, why are we all divided when it comes to building market websites?

I realize that everone has their own way of doing things, their own preferred technologies, and their own ideas on how to build websites, but since we overcame all those things with creating the unified upload format, I was wondering whether we might be able to do it with creating a market website.

We have always been able (for some websites) to contribute to an already existing website, but I have found that the difficulty with doing so is that you feel it is someone else's baby already. They might accept your code, or they might reject it for any number of (valid or invalid) reasons.

I know I have things I'd rather not see in EMK, but I've also seen lots of features on other websites, that I haven't been able to produce myself, or that I think would be awesome, if given a decent layout. I also have lots of ideas on how I would redo a market website if I would start from scratch again, and there are probably lots of developers out there whom would be able to give me very good suggestions on how to do it even better.

Therefore, I pose to you the question: do you think we would be able to build a common market website together, pooling development resources, and quite possibly building the best site yet? If not, why? If yes, how do you propose we go about it?

This is just a question to the community, whether anything comes of it is not an issue yet ;)

I think you may have hit the nail on the head with the differing approaches being the primary motivator. For example, I absolutely loathe PHP (which is my completely unreasonable, flawed, subjective opinion), and my free time has to be spent on things that i enjoy (not PHP). Some want to use Ruby on Rails, others like Scala, and my own tastes gravitate towards Python web frameworks. That's a lot of subjective preferences to reconcile!

As you have noted, working on a market site is a very time-consuming process with a very low probability of long-term survival. There are just too many moving parts for one or two people to pay for, develop, maintain, and continue to run on top of real life, relationships, and other things. Fortunately, we can ease the burden for everyone and better the whole.

One of the huge motivations behind EMDR was to take part of that market data site "stack" and "outsource" it to the point where market site developers don't need to deal with uploader politics and gathering their own swarms of players to submit market data. It doesn't matter what language EMDR is written in, since we have language-agnostic communication between producers and consumers. EMDR is designed to not be a heavy financial burden on me or any of the volunteers, hopefully ensuring a long life. I think these combination of attributes are the potential way to approach sustainable community development. Language-agnosticism, a shared and lesser financial burden for volunteers, and extreme specialization (it only does a few things). And of course, such an effort has to be 100% open source for it to survive long-term.

I think the next steps towards your more community-driven ideal would be to "outsource" additional components that typical market sites re-invent from site to site. You probably know these better than I do, but it seems like a lot of EMDR users aren't sure of the best way to store and analyze market data. Perhaps that would be a good barrier to overcome together as a community. Maybe we write a super-smart and paranoid consumer so others don't have to. Maybe we serve the data in a highly performant, reliable manner (more so than the current multi-purpose market sites can). If we build re-usable, language-agnostic components that developers can plug in, we achieve the community-maintained ideal (since it's mutually beneficial for all of us to improve on them), and we empower everyone.

Small, simple pieces are the key, IMHO. Build a small, simple, component, make it reachable/usable from almost any language, build a community around it, and repeat. Keep doing this and I think we move towards convergence.

Another potentially useful idea would be a community-developed EMDR consumer or market data accepting gateway that did de-duping and validation/cross-checking of incoming data. It could spit JSON back out over ZMQ or write directly to a DB for later aggregation and statistical analysis.

Along that same line, I just released a set of EVE market data structures and serializers (to/from Unified format) here: https://github.com/gtaylor/EVE-Market-Data-Structures

That could be re-used for uploaders, HTTP uploader accepting gateways, things like EMDR, and market sites (consumers). Pretty much no documentation but the unit tests, but hey, release early, release often :)