These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Why are we not combining market data, player devs?
Author
Previous Topic Next Topic
Callean Drevus
Perkone
Caldari State
#101 - 2011-12-04 19:36:17 UTC
I think solarSystemID is not necessary, on the other hand, it's very convenient, as it saves a query to the database every upload :)

Quote:
O.o
This is usually the second step I take for my own projects. I have a server next to me, with Subversion running over mirror raid.


Indeed, I normally also have an offsite backup, but this time it seems I got so caught up in developing the thing that I forgot it (I normally do my copy to live by way of version control, so it's impossible to forget, but desktop applications are packaged).

Developer/Creator of EVE Marketeer
Dragonaire
Here there be Dragons
#102 - 2011-12-04 19:53:00 UTC
Guess it comes down to how normalized your DB is but since orders always happen at stations and they don't move from solar system to solar system there's really no reason to be saving it to the DB and only need to use join on selects which should be being cached in most cases anyway. If you're trying to keep some common per region or solar system stats table up to date it should be ran at the end of the feed insert/update so it's only running once per feed not on each row which would be very slow.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#103 - 2011-12-04 21:17:39 UTC
Dragonaire wrote:
So been think on the JSON order feed stuff a little more and wondering if it really needs to include solarSystemID since that can be inferred from the stationID. The same could be said for the regionID but since it's only used once in the metaData it's less of an issue IMHO.

Anyone else have a feeling about this one way or the other?

The main question is: Do you need it to know at the insertion stage?

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Kaladr
Viziam
Amarr Empire
#104 - 2011-12-04 21:22:28 UTC
I made one minor change: columns is now a list instead of just a string. No reason to split on a tokenized string if you don't have to.

As for solar system, EVE-Central runs with somewhat denormalized tables, and makes use of solarSystemID. But its not a big loss if it doesn't exist.

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Dragonaire
Here there be Dragons
#105 - 2011-12-04 22:01:22 UTC  |  Edited by: Dragonaire
Main reason I changed columns to be a list was to have it match how it was done in the APIs but either way works fine. Having it as an array would be better in that it'll directly match up with rowset rows.

So after making myself look a bit foolish on another thread the next question I have is should 'currentTime' be formated as in the API with'-' and ':'s or like is done in log file names with '.' and nothing?
The API has advantage of never having anyone having the same confusion I did but the other format would let it be extracted directly by cache readers from the file name. It's understandable why CCP didn't use them in file names because they aren't allowed on many OSes or have special means but it purely up to us on the feed.

Edit: Just another couple thoughts on including 'solarSystemID' or not. For API only projects like Yapeal where we'd be extracting the data from the APIs we'd need to do a join to add it since in the APIs we only have stationID. This would raise the bar for other software that might want to do the same thing. It's something I could do in Yapeal but not really something I would want to do just for this new feature I'm looking at adding.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#106 - 2011-12-04 22:32:39 UTC
I'd use one of the standard time formats. Perhaps, ISO 8601 (XML standard date specification, like
2004-02-12T15:19:21+00:00).

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Kaladr
Viziam
Amarr Empire
#107 - 2011-12-04 23:04:14 UTC
Tonto Auri wrote:
I'd use one of the standard time formats. Perhaps, ISO 8601 (XML standard date specification, like
2004-02-12T15:19:21+00:00).


+1 for standard formats. While I like the idea of converging on an existing format, the EVE one isn't exactly the best one :)

Creator of EVE-Central.com, the longest running EVE Market Aggregator
Dragonaire
Here there be Dragons
#108 - 2011-12-04 23:06:30 UTC  |  Edited by: Dragonaire
I do wish CCP had used that for the APIs it really would have made everything easier. It might make sense to do that here and just have everyone manage the conversion in the clients.

So there's another related question here about the rowset 'issueDate'. Should it also be converted so all the date-times are in the same format? I think it would be a good idea and barring that since we do have 'issueDate' in the same format as the APIs then 'currentTime' should be the same as it.

I'm up for everything being in ISO 8601 as it would let clients use their local time in 'currentTime' and the times from the API and cache are already in UTC so they can be converted with simple string appending. I'll update the wiki as I think it's a good way to go with this one.

Edit: wiki is updated for new dates.

I also dropped solarSystemID from 'columns' and 'rowset' but that still leaves a issue for me in Yapeal with 'regionID' Sad So I have another Idea what to do for it and 'solarSystemID'. JSON has a null value so what if we allow both 'regionID' and 'solarSystemID' be null or set? That way clients that have access to them can include them but ones that don't can just use null instead.

Does this sound like a solution that workable for everybody?

Edit : update example to show a row with null in it.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#109 - 2011-12-05 00:24:32 UTC
To answer your question, I can give you two immediate thoughts:
1. Using raw data as coming from cache reader, is easier on processing. But.
2. You are not guaranteed, that the data from cache reading would stay in the same format for foreseeable future.

My own opinion: While using human-readable format for rowset data is tempting, usage of more compact representation is advisable for technical reasons.
I'd stick with UNIXTIME here.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Tonto Auri
Vhero' Multipurpose Corp
#110 - 2011-12-05 00:34:37 UTC
Dragonaire wrote:
I also dropped solarSystemID from 'columns' and 'rowset' but that still leaves a issue for me in Yapeal with 'regionID' Sad So I have another Idea what to do for it and 'solarSystemID'. JSON has a null value so what if we allow both 'regionID' and 'solarSystemID' be null or set? That way clients that have access to them can include them but ones that don't can just use null instead.

Does this sound like a solution that workable for everybody?


I have no JSON experience myself (a regrettable hole in my education, which I have to plug at some point), but I think your reasoning is sound.

On another note, whoever edited the wiki, please check the first example :) And disable spell checking addons while doing so. (volEntered - volunteered)

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Dragonaire
Here there be Dragons
#111 - 2011-12-05 00:50:17 UTC  |  Edited by: Dragonaire
Quote:
On another note, whoever edited the wiki, please check the first example :) And disable spell checking addons while doing so. (volEntered - volunteered)

I don't think it was my edit that did that but I fixed it anyway.

JSON is really easy to understand really the spec for it is like one page. http://www.json.org/

Quote:
My own opinion: While using human-readable format for rowset data is tempting, usage of more compact representation is advisable for technical reasons.
I'd stick with UNIXTIME here.
I'll agree timestamps are more compact but much harder to understand and not as portable. I'd like to see the format to be largely understandable without any docs for even people that didn't read this thread or help with it's development. At most they should just need to be directed to a page like the examples ones and it would clear up anything that might not be totally clear without it. After all it is an exchange/transfer format not really a storage one even though making it easy to work with when trying to store it is important too. That's just not its primary propose.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#112 - 2011-12-05 02:53:35 UTC  |  Edited by: Tonto Auri
While i'm still around, two more issues to throw in a pot:
1. Data encoding. While there's little to no string data communicated in these requests, I think it's important to make a note that any string data SHOULD be encoded in UTF-8.
2. It needs a recommended algorithm to generate 'unique' field value. Something as "simple" as md5 of a certain string value (f.e., program signature concatenated with datetime of a generated request). But not leave it as "oh, just put something there".

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Dragonaire
Here there be Dragons
#113 - 2011-12-05 04:58:35 UTC  |  Edited by: Dragonaire
Quote:
1. Data encoding. While there's little to no string data communicated in these requests, I think it's important to make a note that any string data SHOULD be encoded in UTF-8.
From JSON docs: A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.


Quote:
2. It needs a recommended algorithm to generate 'unique' field value. Something as "simple" as md5 of a certain string value (f.e., program signature concatenated with datetime of a generated request). But not leave it as "oh, just put something there".
Edit: Now I understand where you came up with this someone added 'unique' again after I thought we'd already decided NOT to do something like that since it wasn't needed P

To whomever added it again consider:
If you are talking about a way to prevent network loops it'll be up to each site to decide how they detect it but some pointers on what fields might be used to do this could be added to the wiki. Since this will be used from upload clients to sites or site to site pulls there shouldn't be any need to include something beyond the existing metadata.

In the case of cache readers they already usually delete the cache files after sending so they don't try to resend the same data. For something based on the API you could use the cachedUntil time.

In the case of site to site transfers it'll be done as pulls from RESTful service where the 'client' well be asking for just the updates they want and will need to decide based on the data if they are going to update their internal data. So for example a request for Amarr Shuttle orders in Kor-Azor would be something like www.example.com/EveApiFeed/Kor-Azor/Amarr%20Shuttle/.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Dragonaire
Here there be Dragons
#114 - 2011-12-05 09:46:32 UTC
Ok I made a change to "updateKeys" which should allow people to use the "key" for several different things. Also tried to make it clearer for people what can and can't be used from the JSON message for the "key" here as well. It may still need some refining but at least now everyone that wants to add something 'unique' to the message to ID it can without interfering with anyone else's idea how to do the same thing Blink

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Tonto Auri
Vhero' Multipurpose Corp
#115 - 2011-12-05 13:41:39 UTC  |  Edited by: Tonto Auri
Dragonaire wrote:
From JSON docs: A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.

I did read that. I still insist on clarifying this in format documentation. With a reference to RFC4627 for further information.

A sidenote: You can't use "+" as a substitute for space in URL. You MUST use escape sequence ("%20" in this case).

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Desmont McCallock
#116 - 2011-12-05 13:49:43 UTC
I don't know about you but in .NET if you use HttpUtility.UrlEncode it adds "+" as space.
On the other hand, if you create a new Uri from the string it uses "%20" as space.
Tonto Auri
Vhero' Multipurpose Corp
#117 - 2011-12-05 13:54:18 UTC
Ok, i'm exhausted trying to work around forum stupidity.
@Desmont: unless you encode URL parameters, using "+" as space replacement is just wrong.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison
Desmont McCallock
#118 - 2011-12-05 13:56:57 UTC
Tonto Auri wrote:
Ok, i'm exhausted trying to work around forum stupidity.
@Desmont: unless you encode URL parameters, using "+" as space replacement is just wrong.

I agree.
Dragonaire
Here there be Dragons
#119 - 2011-12-05 16:39:04 UTC
I take it you guys didn't like my example above Blink I'm sure you are right and it should be done as "%20" so I edited it Smile.

So besides my bad URL what does everyone think of the change I made?
I didn't realize there was a problem with how the "uploadKeys" was done before until I was trying to document it and didn't have anything for the Name column in the table. I'd also just finish deleting that whole "unique" thing again and it hit me it might be possible to do something that could be used for either or both without it interfering with anyone else ideas of how these things could be solved. So now we have a "key" where anyone that cares to can either do something neat and useful or just shoot themselves in the foot instead and have minimal impact on anyone else.

A 'BAD' example:
Use "key" to hold to complete encrypted message minus "uploadKeys" and "currentTime" P

'GOOD' example:
Use "key" to hold a simple registered clientID so they can get credit for uploading market order data.

'BETTER' example:
Use "key" to hold clientID like in 'GOOD' but also a digest of it and the complete message from the 'BAD' example.
"key" : "{clientID}|{digest}"

There are many other possibilities but that should give everyone some ideas about how it could be used or abused.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Callean Drevus
Perkone
Caldari State
#120 - 2011-12-06 23:05:34 UTC  |  Edited by: Callean Drevus
Enlighten me again why someone decided to drop the uniqueId? I wasn't the one whom replaced it, but I still advocate it's use.

Instead of implementing a silly hashing function on the server, which admittedly, you could implement any way you want, but which would be a failure by default since you would be reimplementing the very unique uniqueness of a UUID.

I personally very much like the idea of only having to compare a list of recently uploaded id's against the id of a new upload to see if I already got it.

That is, any calculation I can get done clientside is welcome, since that saves resources on the server.

And I like the 'good' example, I'll personally eliminate any upload where the key is longer than 50 characters :P


UPDATE: I saw someone mention on the wiki that a forwarder can modify the 'currentTime' field. It should not, since it messes up the integrity of the data (you really want to know when the message you recieve was actually generated). If it forwards stuff it might be a better idea to implement a different field to keep track of that.

Developer/Creator of EVE Marketeer