These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

POS Tracker - FGV - 5.2.1

Author
Frozen Guardian
The Scope
Gallente Federation
#61 - 2011-12-06 01:02:39 UTC
Thx Drax that may explain some of the setups we couldn't figure out. I'll make sure to add the information for people who want to setup crons.


In other news:

mmmmm fuelblocks

http://iceneko.com/eve/mmmfuelblocks.png

ohh nom nom nom.
Gloria Lena Koraka
State War Academy
Caldari State
#62 - 2011-12-06 03:34:24 UTC
Hi, please include an option to calculate what is needed to build the required amount of fuel block at perfect or optimal ME.
Thanks !
Frozen Guardian
The Scope
Gallente Federation
#63 - 2011-12-06 04:49:53 UTC
Gloria Lena Koraka wrote:
Hi, please include an option to calculate what is needed to build the required amount of fuel block at perfect or optimal ME.
Thanks !


Yeah I had planned on something like that, at least if you had perfect ME. Not sure if I'll have enough time to do more than that at first like different ME levels or allow you to calculate on the fly. But I hear you can get perfect ME right now for the FB BPOs before its release so not sure if that many people would care anyway. Not even sure why CCP even thought making people doing ME research on the fuel blocks was needed. Just seems dumb and pointless imo.
Gloria Lena Koraka
State War Academy
Caldari State
#64 - 2011-12-06 15:32:07 UTC
Great, keep up the good job, as long as we have perfect ME we would be happy.
Kismeteer
Bat Country
Pandemic Horde
#65 - 2011-12-06 16:12:09 UTC
Gloria Lena Koraka wrote:
Hi, please include an option to calculate what is needed to build the required amount of fuel block at perfect or optimal ME.
Thanks !


These are well known numbers. For 5% waste products, you divide the max usage by 10.5. For 10% waste products, you go with a 5.5% number. Take the largest number in a fuel block, 420 isotopes, and do the math so 420/10.5 = 40. ME 40 is a perfect build. If you want to just save all oxygen, ME 2 is fine. If you want to save all HW and LO3, 158/10.5 = ME 15 (rounded, CCP adds .5 and then truncates).

ME 15 is like a day and 4 hours. Also, Time Efficiency is totally worth it on these prints. With perfect skills and no build bonus, PL 0 is 4 minutes. With a good facility, at PL 3 you're looking at a build time of 2 minutes 6 seconds, PL 10 is 1:55, and PL 14 is 1:53. So I recommend you get at least a little Time Efficiency on there, another 6 hours maybe, and lots of empire slots.

Finally, blocks are finally able to be put in towers, but the API will still not return blocks because the API team didn't get the memo. Hopefully CCP will resolve this soon.
Gloria Lena Koraka
State War Academy
Caldari State
#66 - 2011-12-06 17:11:59 UTC
Kismeteer wrote:

snip ...lots of number...


I'm not saying I don't know what these number are, I'm saying I want the tool to tell me I need x amount of minmatar block and y amount of amar block to fuel everything for the specified number of days which break down to Z number of material needed to build these rounded up block number so I don't have to do all the math myself of how much of each material I need to fuel all the tower, like the tool actually does it create you a shopping list (totals fuel for all tower). So the buyer goes and buy all that is required and bring it for the manufacturer who build the specified amount of block and the fuel technician moves these block into each tower. Easing the life of everyone.
Kismeteer
Bat Country
Pandemic Horde
#67 - 2011-12-06 17:24:32 UTC
Gloria Lena Koraka wrote:
Kismeteer wrote:

snip ...lots of number...


I'm not saying I don't know what these number are, I'm saying I want the tool to tell me I need x amount of minmatar block and y amount of amar block to fuel everything for the specified number of days which break down to Z number of material needed to build these rounded up block number so I don't have to do all the math myself of how much of each material I need to fuel all the tower, like the tool actually does it create you a shopping list (totals fuel for all tower). So the buyer goes and buy all that is required and bring it for the manufacturer who build the specified amount of block and the fuel technician moves these block into each tower. Easing the life of everyone.


Then it's pretty simple: research an ME 40 BPO and he's said he'll add a fuel total in there. Done. No loss at all, easy math for our pos app, everyone wins. Just my opinion, but this is a pos fueling app, not a production application, don't make it too complicated.

If you want easy research, throw up a lab for a bit, or just research it in low sec as there are lots of open, very cheap slots.
Gloria Lena Koraka
State War Academy
Caldari State
#68 - 2011-12-06 20:15:01 UTC
/rant on

Gloria Lena Koraka wrote:
Great, keep up the good job, as long as we have perfect ME we would be happy.


Maybe you should start reading what I wrote just before your first post...

oh wait, sorry I forgot you are a member of goons and that's a prerequisite
/rant off
Kismeteer
Bat Country
Pandemic Horde
#69 - 2011-12-07 19:09:12 UTC
Gloria Lena Koraka wrote:
/rant on
Maybe you should start reading what I wrote just before your first post...

oh wait, sorry I forgot you are a member of goons and that's a prerequisite
/rant off


Yeah, that was pretty 'goon like', giving you perfect ME numbers because addition is hard.
Kismeteer
Bat Country
Pandemic Horde
#70 - 2011-12-09 16:32:25 UTC
Good news, they announced the date on the energon cube cut over.

Bad news, it's Jan 24th. So a month and a half.

Here's hoping you are still procrastinating 4 days before original timeline.

https://forums.eveonline.com/default.aspx?g=posts&t=44269
Frozen Guardian
The Scope
Gallente Federation
#71 - 2011-12-09 18:01:39 UTC  |  Edited by: Frozen Guardian
Kismeteer wrote:
Good news, they announced the date on the energon cube cut over.

Bad news, it's Jan 24th. So a month and a half.

Here's hoping you are still procrastinating 4 days before original timeline.

https://forums.eveonline.com/default.aspx?g=posts&t=44269


Well okay then lol. Well this kinda sucks in a sense because some of the patch notes could come out earlier but not anymore since they are all attached to the fuel blocks patch. Would be too many changes to reverse. Ohh well, expect the fuel block patch probably the week/weekend before January 24th. Gives me more time though so I'll probably add in some additional fixes or updates too. This may turn out to be one hell of a patch.

...or I'll wait till the 23rd...before working on it some more. bwhahaah :P
Kismeteer
Bat Country
Pandemic Horde
#72 - 2011-12-09 21:19:23 UTC
Frozen Guardian wrote:
Ohh well, expect the fuel block patch probably the week/weekend before January 24th. Gives me more time though so I'll probably add in some additional fixes or updates too. This may turn out to be one hell of a patch.

...or I'll wait till the 23rd...before working on it some more. bwhahaah :P


I'd recommend patching the fuel block detection code soon, as they might just double the size of pos's, and so the number of fuel blocks might get to be important. Make it an uncomplicated patch, I guess. Then when it starts burning cubes, that's an entirely different ball of wax. `8r/

Good luck man. The API is a pain in the ass.
Frozen Guardian
The Scope
Gallente Federation
#73 - 2011-12-10 02:10:37 UTC
Kismeteer wrote:
Frozen Guardian wrote:
Ohh well, expect the fuel block patch probably the week/weekend before January 24th. Gives me more time though so I'll probably add in some additional fixes or updates too. This may turn out to be one hell of a patch.

...or I'll wait till the 23rd...before working on it some more. bwhahaah :P


I'd recommend patching the fuel block detection code soon, as they might just double the size of pos's, and so the number of fuel blocks might get to be important. Make it an uncomplicated patch, I guess. Then when it starts burning cubes, that's an entirely different ball of wax. `8r/

Good luck man. The API is a pain in the ass.


Well I had a reply and then the forums didn't actually post it so...yeah now I hate these new forums again.
Kismeteer
Bat Country
Pandemic Horde
#74 - 2011-12-13 15:41:16 UTC
Just a heads up, there was a suggestion that they double the fuel bay temporarily, and it was implemented.
https://forums.eveonline.com/default.aspx?g=posts&m=494033#post494033

BTW, that guy is a goon as well, it's his market alt.
Solo Drakban
Goosefleet
Gooseflock Featheration
#75 - 2011-12-16 15:40:54 UTC
Let's see if the forums eat this post.. (Seriously CCP, these are forums. There are hundreds of well written forums out there, how can you manage to have such horrible forums that previewing a post manages to destroy the post. Also, no [pre] or [code] tags? Seriously? :argh:)

I've been looking at optimizations for this app after I noticed that the track.php was taking a long time to load and that my database was screaming when it was loading. Poking into the query logs I found this query which seems to be called 2-3 times per tower:

SELECT *
FROM update_log
WHERE type_id = '52'
AND type = '1'
ORDER BY id DESC;

Ignoring the SELECT *, I ran an explain on this and noticed that there were no indexes (update_log_backup is just a duplicate of our update_log, I created it so I could play without killing existing data):

mysql> explain SELECT * FROM update_log_backup WHERE type_id = '52' AND type = '1' ORDER BY id DESC;
+----+-------------+-------------------+------+---------------+------+---------+------+--------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+------+---------------+------+---------+------+--------+-----------------------------+
| 1 | SIMPLE | update_log_backup | ALL | NULL | NULL | NULL | NULL | 454823 | Using where; Using filesort |
+----+-------------+-------------------+------+---------------+------+---------+------+--------+-----------------------------+

The query itself generates 2,799 rows and takes 0.23 and does so by doing a full table scan of almost 500k rows. Shocked

Time for indexes!

mysql> create index ULstuff on update_log_backup ( type, type_id );
Query OK, 0 rows affected (4.62 sec)
Records: 0 Duplicates: 0 Warnings: 0

mysql> explain SELECT * FROM update_log_backup WHERE type_id = '52' AND type = '1' ORDER BY id DESC;
+----+-------------+-------------------+------+---------------+---------+---------+-------------+------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+------+---------------+---------+---------+-------------+------+-----------------------------+
| 1 | SIMPLE | update_log_backup | ref | ULstuff | ULstuff | 12 | const,const | 2798 | Using where; Using filesort |
+----+-------------+-------------------+------+---------------+---------+---------+-------------+------+-----------------------------+

Hey, an index to read from, no more table scans! This reduced the query from 0.23 seconds to 0.01 seconds and returns the same number of rows. This took the load time of track.php from ~50 seconds to ~4 seconds.

It also looks like the function doing this query is only using the first row to determine the last update for a POS... If this is correct you may want to consider using a limit in there and only returning a single row instead of all of them..

Your query:

mysql> SELECT * FROM update_log_backup WHERE type_id = '52' AND type = '1' ORDER BY id DESC;
+--------+--------+---------+------+--------------------------------+------------+
| id | eve_id | type_id | type | action | datetime |
+--------+--------+---------+------+--------------------------------+------------+
| 454189 | 0 | 52 | 1 | EVEAPI XML STARBASE API UPDATE | 1323824523 |

With a limit:

mysql> SELECT * FROM update_log_backup WHERE type_id = '52' AND type = '1' ORDER BY id DESC limit 1;
+--------+--------+---------+------+--------------------------------+------------+
| id | eve_id | type_id | type | action | datetime |
+--------+--------+---------+------+--------------------------------+------------+
| 454189 | 0 | 52 | 1 | EVEAPI XML STARBASE API UPDATE | 1323824523 |
+--------+--------+---------+------+--------------------------------+------------+

Same data, 2,798 less rows.

You could also get the same result without looping over the list of POSes by doing a correlated sub-query or a self-join. In fact it should be possible to generate the entire from page from a single, wel optimized query (excluding pagination and other misc queries, I'm talking about just the table) with a few joins and possibly sub-selects.

I have other suggestions for optimizations after going through the code and the DB structure, but I'll hold off until I know if they'll be useful or not.. :)
Frozen Guardian
The Scope
Gallente Federation
#76 - 2011-12-18 03:34:04 UTC  |  Edited by: Frozen Guardian
When I notice things like this I just shake my head. I found several things that could been easily fixed or wrote differently since the start of the tracker and just noticed them now during the re-write for Fuel Blocks. Though I'm not an expert by any means, I've just seen some "interesting" ways about doing some DB functions and the related. Probably wrote some myself just to get stuff to work asap though I try to avoid it the best I can and I usually test a lot to make sure the function works how I want it.

So I absolutely love it that you noticed this as I've gone past this function several times and never noticed it. I had no need to look at it directly and never even knew it was pulling in all rows. Just dang son, dang. I don't have a huge tower list that gets used like crazy to work from so it's much harder for me to notice these things. Especially since my queries complete in 0.0001. The end result though is that the hoursago function has been updated with a limiter and index since that's easy to do and tested and works wonderfully. This should also speed up various other functions that use hoursago even just a little bit. The update log itself needs some work in terms of how its updated, what data gets put into it, and what isn't being put into it that should be. Especially since I want a nice viewer for admin accounts to at least be able to see certain things. That's for later though :)

But if you have any further optimizations or notice anything that you think should be handled differently please let me know in any way shape or form you want. EVEMail, Forums, Google Project, or Email. Yes I find these useful.

I'm open for ideas especially when people put the solutions and reasons in front of me.

Thanks Solo Drakban! <3

P.S. I like Kismeteer more though due to her avatar. :P
Solo Drakban
Goosefleet
Gooseflock Featheration
#77 - 2011-12-19 15:22:55 UTC
Another quick thing:

In track.php at line 104 there is:

$row2 = $posmgmt->GetLastPosUpdate($row['pos_id']);

$row2 never seems to get called before the end of the function nor in the template. I commented this out and nothing seems to have broken and there is no reduced information on the tracking page as far as I can tell, but it is saving on some extra queries to the update_log table. :)

Frozen Guardian
The Scope
Gallente Federation
#78 - 2011-12-19 22:23:54 UTC
Solo Drakban wrote:
Another quick thing:

In track.php at line 104 there is:

$row2 = $posmgmt->GetLastPosUpdate($row['pos_id']);

$row2 never seems to get called before the end of the function nor in the template. I commented this out and nothing seems to have broken and there is no reduced information on the tracking page as far as I can tell, but it is saving on some extra queries to the update_log table. :)



Thanks! Yes I see no use for it either and can't seem to find reason why it existed other than it probably was copy and pasted by someone from view/editpos.

In other news, another forum reply gone due to "sorry we've been ganked." I got to remember to write out my replies elsewhere first and paste them into here.
Johnathan Roark
Quantum Industries
#79 - 2012-01-02 07:20:00 UTC
Frozen Guardian wrote:
Solo Drakban wrote:
Another quick thing:

In track.php at line 104 there is:

$row2 = $posmgmt->GetLastPosUpdate($row['pos_id']);

$row2 never seems to get called before the end of the function nor in the template. I commented this out and nothing seems to have broken and there is no reduced information on the tracking page as far as I can tell, but it is saving on some extra queries to the update_log table. :)



Thanks! Yes I see no use for it either and can't seem to find reason why it existed other than it probably was copy and pasted by someone from view/editpos.

In other news, another forum reply gone due to "sorry we've been ganked." I got to remember to write out my replies elsewhere first and paste them into here.



Or it was needed at one point, but something else changed, and never got removed. I'm guessing you can't find anyone that could explain 100% why every line was put in. Its been worked on by so many people now and changed with pos updates. If anyone wonders why I stopped working on it, it needs to be rewritten from scratch, but I didn't have the time. Fixing bugs became such a pain due to how complex it had gotten. Frozen is doing and excellent job at it though.

EVEVERIFY - A recruiting API Verification and Audit Tool

Also try out Yapeal for your php api needs

Gloria Lena Koraka
State War Academy
Caldari State
#80 - 2012-01-09 17:35:39 UTC
Frozen Guardian, do you plan to release an update for the fuel block a bit in advance so that we can update our pos tracker installation before the switchover so that we have time to plan with the new fuel block requirement in advance.

Thanks !