These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Note on some broken CREST things

First post First post
Author
Previous Topic Next Topic
salacious necrosis
Garoun Investment Bank
Gallente Federation
#41 - 2016-06-17 01:37:53 UTC
Bump. Only to see if we can induce Fox to post another meme.

Use EveKit ! - Tools for EVE Online 3rd party development
Pirate Lord Kane
Doomheim
#42 - 2016-06-17 04:37:44 UTC  |  Edited by: Pirate Lord Kane
Thank you for the updates, Enjoy your weekend!

To all the annoyed players, it's not the end of the friggen' world my corp is missing several kills that my newest member is proud of getting solo... But you know what I can still pull them up in corp kills and praise him for it everything is flowing in game. If anybody doubts the validity of your loss for SRP tell him to check corp losses, if anybody really needs to be trolled with their 500 million isk phantasm they lost to my destroyer yesterday (you know who you are) I'd be happy to post it in local again.

TY FoxFour for keeping us up to date and the detail you went into explaining the situation, Players... Try actually using in game resources instead of relying on third party so much... Be smarter than the bug.



P.S. Second what the guy above me said about more FoxFour memes.
Evan Giants
Native Freshfood
Minmatar Republic
#43 - 2016-06-17 04:40:38 UTC
Man, some people are brutal. Let devs have their breaks! They've been working hard years for players.
Squizz Caphinator
The Wormhole Police
#44 - 2016-06-17 05:15:22 UTC  |  Edited by: Squizz Caphinator
CCP FoxFour wrote:
Squizz Caphinator wrote:
Rear Admiral Barrington wrote:
But without the ability to compare epeens on ZKill, the whole of FW space is going to become a pacifist nightmare this weekend!

We'll all be holding hands and singing Kumbaya before Saturday is out!


I might have a fix in place that will get around this issue. Won't be able to implement until tomorrow morning though.


Color me intrigued. Looking forward to seeing what you come up with.


Alrighty, so the majority of the mails come through via XML api which contain all of the information that CREST has with the exception of the warID, if applicable. So I wrote some code that converts the XML format to the CREST format for the killmails that are having issues, then plopped the converted data into the right collections and reset the process flag for that killmail.

It's a hack but gets the job done. I think I can live without having the war information displayed for these mails in the meantime. Come next Tuesday when the overall CREST issues get corrected I will reset all these mails and have them reparsed just to be on the safe side.

Here's a short example of the code in action:

Quote:

2016-06-17 05:06:57 > 54648021 -> 500 http errorCode
2016-06-17 05:06:58 > 54647859 -> 500 http errorCode
2016-06-17 05:07:01 > XML->CREST conversion for 54648078 completed
2016-06-17 05:07:01 > XML->CREST conversion for 54648077 completed
2016-06-17 05:07:01 > XML->CREST conversion for 54648021 completed
2016-06-17 05:07:01 > XML->CREST conversion for 54647859 completed


I haven't noticed any odd side effects yet, but it's late and I've had a couple of drinks (don't tell the woman) and I just chugged through this code. And yea, I couldn't wait till morning. I'm on a creative high tonight after finishing my graduate research paper Shocked

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Ohno no Borrox
xX-Crusader-Xx
Tactical Narcotics Team
#45 - 2016-06-17 05:19:53 UTC
Ayuren Aakiwa wrote:
Ohno no Borrox wrote:
Man, I just love it when CCP paint a big target on their own backs.

So what you're telling us is that, although this issue has been in around for at least two days, it's taken you this long to get it fixed. Additionally you do not have enough confidence in the "fix" to deploy it anyway (if it is really fixed). Further, why do it Monday, so you can fix any more bugs in it that may jump out and bite you, when you can leave it until Tuesday when it will be lost in the pile of other things that go wrong - it's not like the users are customers or anything.

I really want to work there - it's just so customer focused




mate do you honestly care that much about looking at a killboard? You should go outside or maybe even just upstairs I'm sure your mom misses you (she told me so last night)


Well you must have been having a seance then, 'cos she's a teeny bit dead.
Ohno no Borrox
xX-Crusader-Xx
Tactical Narcotics Team
#46 - 2016-06-17 05:26:30 UTC  |  Edited by: Ohno no Borrox
CCP FoxFour wrote:
Ohno no Borrox wrote:
you don't address the issue of not getting the fix out on Monday


Assuming you want a real answer and are not trolling, which I will give the benefit of the doubt on here, I am happy to answer that.

Essentially we have a few branches at play in our source control. Lets call the first branch TQ. This branch is meant to be exactly the same code that is deployed to TQ at all times. Lets call our second branch staging. This is where all code goes for some testing and before going to the TQ branch.

I have deployed all the fixes to the staging branch and they have been tested and verified as fixed. However since they are not going out right away I have not ported them to the TQ branch. If I ported them to the TQ branch now and we didn't deploy them until Monday the TQ branch would NOT represent what TQ actually was. This may seem like a stupid little thing that makes no sense and is just delaying code going out but lets look at an example of why this is important.

Lets say I Saturday comes along and we find a critical issue in EVE. Maybe a security vulnerability or something. The issue is important enough that we call someone into the office, they make a fix in the TQ branch, take TQ down, and deploy the new build. Well now my code has gone out as well and I wasn't around. What if my code breaks something? What if my code interacts with the other fix in an odd way? Plenty of reasons to have that branch be clean.

So since I am not porting my code from staging to TQ until as close as possible to the point of us actually shipping it lets look at Monday. To get it in for Monday I have to submit the code the day before to the TQ branch so our automated builds and tests can run. The day before is a Sunday. Fortunately for me CCP is an awesome company that understands a work life balance and encourages us to not work on the weekends if at all possible.

TL;DR: No deployment on Monday because I like having a life and not working on a Sunday.

Hope that answers your question. :)


That seems a reasonable response to the Monday issue. [edit] <<== was my initial response, but then I re-read your answer and guess who you're not putting first: the customer (i.e. the person who pays your wages)

And no, I was not trolling. What I don't understand is why it was not properly tested in hte first place. Just because Microsoft made a habit of releasing software before it was ready does not mean everyone else should

I am not dissing you personally, but I am having a go at CCP for continually releasing buggy updates.

And for the record, and for those that asked - I used to write code, did so for 35 years before I retired. Won't say I never got it wrong, but when I did, I got a rocket up my bottom and no weekend off until it was fixed
CCP FoxFour
C C P
C C P Alliance
#47 - 2016-06-17 07:06:08 UTC
MachineOfLovingGrace wrote:
Sgt Ocker wrote:
So basically what you just said is; I fixed the code AND tested it but it might still break something when implemented. So it wasn't really tested all that well to start with - was it?

CCP doesn't seem to understand - Customer satisfaction should come first, second and third as a priority - If staff having an extra day off is so important, let them have it once the problem is fixed.

There is nothing "awesome" about a company that sees customer satisfaction as such a low priority.

Quote:
TL;DR: No deployment on Monday because I like having a life and not working on a Sunday
Why not? You've already had Friday and Saturday off - That's more of a weekend than a lot of us ever get. Just as well you aren't in a job that requires some commitment, houses would burn, patients would die, etc.
Yes I am comparing your work ethic to the thousands who pay to play Eve.

PS; your TL;DR makes you look bad. Would have been better had you NOT included it.

Because we luv Eve, we will wait while you ponce around "having a life", you poor self entitled thing you.


I am amazed that any dev is still posting in this forums - sometimes I wonder what this community is on about.

Thanks for the insight into the testing/deploying process, FoxFour. How formalized/automated is the deploying, is it basically a button in a webform or an arcane web of scripts?


We have a growing number of automated tests. These range from testing if the server starts to regular python unit tests. The art department also has a bunch of tests they run as well for things like performance testing so we can see if performance changes over time. Anyways, point is those are all automated.

We then also have a large number of automated build jobs that monitor specific files and run builds if those files change and check in the resulting artefacts.

Beyond that we than have a daily build job that runs every day at 22:00 for each branch. This can also be triggered manually when desired. For example during the day when we want to update Sisi with some changes from the morning we will run the build and deploy that.

As for deploying it really depends on the server. All our smaller test servers we can deploy updated pretty much with a couple of clicks. For larger test servers such as Sisi this takes a little more effort. Usually a manual shut down, open special web page and upload build, click deploy build, run any database updates, and click the start cluster button.

TQ is a completely different beast that unfortunately is stuck a bit in the past and has a fair bit of manual steps for deploying to. Generally requires a few people from ops, at least a DBA, and preferably some others on hand.

That is all for the EVE sol/proxy/client code. We have a growing number of other services (image server, SSO, VGS, paparazzi, NGINX servers, sphinx, etc.) that are all built and deployed differently. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#48 - 2016-06-17 07:12:05 UTC
RoomUnaut wrote:
Handle business FoxFour! Software development ain't easy, and always a bad idea to release on a Friday (especially with a well deserved 3 day weekend coming up)!

Thanks for letting us know, and hope you have an epic weekend!

Cheers!
http://33.media.tumblr.com/a23fb6be88127e8bb6fa36a63786b331/tumblr_nj08kejyUd1smcbm7o1_500.gif


You're welcome good sir. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#49 - 2016-06-17 07:13:10 UTC
salacious necrosis wrote:
Bump. Only to see if we can induce Fox to post another meme.


[img]https://media2.giphy.com/media/4fSIakWpZeQqA/200w.gif[/img]

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#50 - 2016-06-17 07:16:19 UTC
Pirate Lord Kane wrote:
Thank you for the updates, Enjoy your weekend!

To all the annoyed players, it's not the end of the friggen' world my corp is missing several kills that my newest member is proud of getting solo... But you know what I can still pull them up in corp kills and praise him for it everything is flowing in game. If anybody doubts the validity of your loss for SRP tell him to check corp losses, if anybody really needs to be trolled with their 500 million isk phantasm they lost to my destroyer yesterday (you know who you are) I'd be happy to post it in local again.

TY FoxFour for keeping us up to date and the detail you went into explaining the situation, Players... Try actually using in game resources instead of relying on third party so much... Be smarter than the bug.



P.S. Second what the guy above me said about more FoxFour memes.


You're welcome. :) No meme or gif for you :P

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#51 - 2016-06-17 07:17:43 UTC
Squizz Caphinator wrote:
CCP FoxFour wrote:
Squizz Caphinator wrote:
Rear Admiral Barrington wrote:
But without the ability to compare epeens on ZKill, the whole of FW space is going to become a pacifist nightmare this weekend!

We'll all be holding hands and singing Kumbaya before Saturday is out!


I might have a fix in place that will get around this issue. Won't be able to implement until tomorrow morning though.


Color me intrigued. Looking forward to seeing what you come up with.


Alrighty, so the majority of the mails come through via XML api which contain all of the information that CREST has with the exception of the warID, if applicable. So I wrote some code that converts the XML format to the CREST format for the killmails that are having issues, then plopped the converted data into the right collections and reset the process flag for that killmail.

It's a hack but gets the job done. I think I can live without having the war information displayed for these mails in the meantime. Come next Tuesday when the overall CREST issues get corrected I will reset all these mails and have them reparsed just to be on the safe side.

Here's a short example of the code in action:

Quote:

2016-06-17 05:06:57 > 54648021 -> 500 http errorCode
2016-06-17 05:06:58 > 54647859 -> 500 http errorCode
2016-06-17 05:07:01 > XML->CREST conversion for 54648078 completed
2016-06-17 05:07:01 > XML->CREST conversion for 54648077 completed
2016-06-17 05:07:01 > XML->CREST conversion for 54648021 completed
2016-06-17 05:07:01 > XML->CREST conversion for 54647859 completed


I haven't noticed any odd side effects yet, but it's late and I've had a couple of drinks (don't tell the woman) and I just chugged through this code. And yea, I couldn't wait till morning. I'm on a creative high tonight after finishing my graduate research paper Shocked


You're a god amongst people friend! Very well done and creative solution. :D

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#52 - 2016-06-17 08:14:16 UTC
Ohno no Borrox wrote:
CCP FoxFour wrote:
Ohno no Borrox wrote:
you don't address the issue of not getting the fix out on Monday


Assuming you want a real answer and are not trolling, which I will give the benefit of the doubt on here, I am happy to answer that.

Essentially we have a few branches at play in our source control. Lets call the first branch TQ. This branch is meant to be exactly the same code that is deployed to TQ at all times. Lets call our second branch staging. This is where all code goes for some testing and before going to the TQ branch.

I have deployed all the fixes to the staging branch and they have been tested and verified as fixed. However since they are not going out right away I have not ported them to the TQ branch. If I ported them to the TQ branch now and we didn't deploy them until Monday the TQ branch would NOT represent what TQ actually was. This may seem like a stupid little thing that makes no sense and is just delaying code going out but lets look at an example of why this is important.

Lets say I Saturday comes along and we find a critical issue in EVE. Maybe a security vulnerability or something. The issue is important enough that we call someone into the office, they make a fix in the TQ branch, take TQ down, and deploy the new build. Well now my code has gone out as well and I wasn't around. What if my code breaks something? What if my code interacts with the other fix in an odd way? Plenty of reasons to have that branch be clean.

So since I am not porting my code from staging to TQ until as close as possible to the point of us actually shipping it lets look at Monday. To get it in for Monday I have to submit the code the day before to the TQ branch so our automated builds and tests can run. The day before is a Sunday. Fortunately for me CCP is an awesome company that understands a work life balance and encourages us to not work on the weekends if at all possible.

TL;DR: No deployment on Monday because I like having a life and not working on a Sunday.

Hope that answers your question. :)


That seems a reasonable response to the Monday issue. [edit] <<== was my initial response, but then I re-read your answer and guess who you're not putting first: the customer (i.e. the person who pays your wages)

And no, I was not trolling. What I don't understand is why it was not properly tested in hte first place. Just because Microsoft made a habit of releasing software before it was ready does not mean everyone else should

I am not dissing you personally, but I am having a go at CCP for continually releasing buggy updates.

And for the record, and for those that asked - I used to write code, did so for 35 years before I retired. Won't say I never got it wrong, but when I did, I got a rocket up my bottom and no weekend off until it was fixed


The thing you are missing is that no matter how much I test it locally or on a test server nothing matches TQ. TQ is on different hardware in a different configuration. On top of that we are deploying a monolithic application that is about 16+ years old. No matter how much testing we do there is always a chance we will make things worse. This isn't a concept strictly restricted to EVE. Pretty much everyone in the modern era wont deploy code on a "Friday" (day before a day off). The biggest difference is we get one deployment a day, downtime, versus most people getting to deploy as much as they want.

So again, it has nothing to do with how much testing we have done or the level of confidence we have in our fix. Things can and do go wrong when deployed to a different environment so we don't take the risk.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Rhazjin
Native Freshfood
Minmatar Republic
#53 - 2016-06-17 10:19:07 UTC  |  Edited by: Rhazjin
Foxfour. You have responded more than any dev. has in recent memory. This is all i have been asking for. A response. Because of this I give you mad kudo's as well as offering you ccp phantoms job. Twisted Communication is extremely important, and you seem to understand that. Thankyou! Crest being down is an issue, but its nothing major. Have a good weekend.
CCP FoxFour
C C P
C C P Alliance
#54 - 2016-06-17 11:17:36 UTC
Rhazjin wrote:
Foxfour. You have responded more than any dev. has in recent memory. This is all i have been asking for. A response. Because of this I give you mad kudo's as well as offering you ccp phantoms job. Twisted Communication is extremely important, and you seem to understand that. Thankyou! Crest being down is an issue, but its nothing major. Have a good weekend.


You too man. Have a good weekend.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Kaoraku Shayiskhun
Simple Designs
#55 - 2016-06-17 11:32:49 UTC
Evan Giants wrote:
Man, some people are brutal. Let devs have their breaks! They've been working hard years for players.


This is the only one, what I really do not understand. They do their job, for money. Why it is something, what you feel you have to thank them? Usually this is the normal way of things.

Btw, have a good weekend, rest enough to fight with the not starting servers after the update Lol
neggies
Deep Core Mining Inc.
Caldari State
#56 - 2016-06-17 13:45:36 UTC
What's it like to push the start button on an entire universe?

Thankyou for the explanation and replying to all these messages <3
CCP FoxFour
C C P
C C P Alliance
#57 - 2016-06-17 13:53:12 UTC
neggies wrote:
What's it like to push the start button on an entire universe?

Thankyou for the explanation and replying to all these messages <3


Kinda disappointing actually :(

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Aineko Macx
#58 - 2016-06-17 15:16:04 UTC
There's more broken stuff:
Trying to pull TournamentStaticSceneData or TournamentRealtimeMatchFrame throws 500 errors:
Unexpected exception thrown of type 'exceptions.KeyError', key:unexpectedException, exceptionType:InternalServerError, refID: ac73554d-221a-4e0c-ae2f-7a49a98a2dda

iveeCore 3.0: The PHP engine for industrial activities and CREST library
CCP FoxFour
C C P
C C P Alliance
#59 - 2016-06-17 15:28:14 UTC
Aineko Macx wrote:
There's more broken stuff:
Trying to pull TournamentStaticSceneData or TournamentRealtimeMatchFrame throws 500 errors:
Unexpected exception thrown of type 'exceptions.KeyError', key:unexpectedException, exceptionType:InternalServerError, refID: ac73554d-221a-4e0c-ae2f-7a49a98a2dda


Will pass that on to CCP Bartender.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Ix Method
Doomheim
#60 - 2016-06-17 15:52:04 UTC
CCP FoxFour wrote:
CCP Bartender

...

Who knew I could love CCP more?

Travelling at the speed of love.