These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
123Next page
 

Big EvE website project: please tell me tips on deployment to prod...

First post
Author
Vaerah Vahrokha
Vahrokh Consulting
#1 - 2014-03-07 16:56:57 UTC
Hello,

I am asking about this here, because this is an EvE related project and because I know a lot of EvE players are competent developers.


I need to setup a large-ish EvE website on a VPS. I know and have done in the recent past the whole "Create Ubuntu VPS from scratch, install everything from daemons, PHP, MySQL, Nginx" (yeah! Twisted) and so on and on.
I have also coded in many languages including PHP, Python (a bit) etc. But always alone, always "develop => deploy => done forever".

What I am missing is this: this time I might not be the lone developer and I need to setup a development environment that can be brought to production.

That is, I'd like to develop stuff on say NetBeans (or other editor), upload it to the server in a development environment and test it out.

Once it passes the unit tests I'd like to start some script or anything and see the development version of the website "magically" update over the production one, possibly also update the MySQL database and end up with a working production site.

I am asking you, the Estimated EvE Developers for hints about how to get this done in a reasonable and as much automated way.

The project will have for sure:

- MySQL database
- A lot of PHP code
- Possibly Python code
- Possibly Mono code
- Possibly some C code

I am expecially interested into the PHP side of things, since it'll make for 90% of the website.


Thanks in advance.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#2 - 2014-03-07 17:18:21 UTC
composer will help a great deal.

Along with storing your code in Git.

Build it in libraries which you include into your main project with composer, with the referencing being handled by packagist so you can run a 'composer update' to grab the latest release versions of the library (but not the trunk)

Then deployment of the main code is switching the git tag, and updating to it.

Automagically updating the database will be painful

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Devilen
Deep Core Mining Inc.
Caldari State
#3 - 2014-03-07 17:29:04 UTC
I agree with Steve with everything he said.
I am working on a few different projects in and out of Eve and composer is a good start and git is a must have in my book. The database is going to be a pain if you are wanting to do it automagically only because of all the things that could go wrong you would take forever to program for them but it is not impossible.

First thing is find out what you want to create and then work on it and go at a pace you can maintain is the last thing I could advise on this matter. You could also see about joining any of the projects currently in progress if anyone is looking for help but that is all up to you.
Vaerah Vahrokha
Vahrokh Consulting
#4 - 2014-03-07 19:08:22 UTC  |  Edited by: Vaerah Vahrokha
Thank you to both of you!

I am quite inexperienced in the git / composer stuff. The last time I have used a source code control, it has been SVN.

There's a missing link in what you said (or maybe not, I just don't see it): the testing.

I mean, at the moment I am doing this:

1) I write code in my Windows box. I don't have a development environment, because I have many daemons and processes that are hard to replicate in Windows. I code from different computers in different locations. Such is life when you are a RL securities trader who develops at home but also on the beach and wherever else P

2) So when I save, NetBeans uploads the code to a (series of) folders on the Ubuntu box.

3) These folders are the production tree so I am basically coding on the "live" website. In reality it's not live yet, since the website is not published anywhere.

4) I run unit tests (also in a folder in there), mainly through PHPUnit, they test the production tree.

5) Once tests pass I can see the website by going to the ip address or FQDN of the website.


And I need to switch to this:

1) Write code like at point 1 above. Possibly done by more than 1 person.

2) The code goes in a development tree. I suppose I can make a sister directory to "public_html" called "dev_html" or similar and make the saved code go there.

3) I need to make the development code point to a development database (I won't also spawn a dedicated development database server instance).

4) Once I have saved my stuff on that development tree I need a way to make it "show" in something like: "dev.domain.tld". I will put a password on it of course.

Basically I want to see the stuff working, BEFORE I make it merge onto the production version.

5) Only then, I would use something (you suggest git) to deploy development to production.


How do I cover all these steps? Am I missing something?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#5 - 2014-03-07 19:31:54 UTC
The main difference between git and svn, is that you don't check out a particular revision from git. You clone the /entire/ repository. So you can do all your commits locally, then push them when you're happy, to the central copy.



If what you're doing is open source, take a look at https://scrutinizer-ci.com/ which can do some of the code coverage steps, as well as making sure you're sticking to the standard you've set. php mess detector is also handy, as is phpcs for local code standard checks.


Most of my coding is done in vim, using ssh to access my server.


composer is pretty simple to work with. you define your composer.json file, which will grab down the appropriate libraries, and create your module loader. Dependency management is always nice Smile


Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Devilen
Deep Core Mining Inc.
Caldari State
#6 - 2014-03-07 19:56:13 UTC
Well lets see if I might be able to help clear this up for you.

Testing - I have 2 types of sites with my work. One is the main (Live site) the other is a testing site. Most my testing sites are a subdomain off main my domain e.g. dev.example.com. The way this gets tested is everyone who goes to it knows the data could be deleted/removed at any time without reason.

Currently at the moment I see your saying you write code on many devices or just one? Because I personally use 4 devices all of them are using Git so you can imagine all the commits I have in my git Repo. I am not a fan of netbeans but if you understand it awesome! I use Sublime text 2 and that helps me keep my environment from one device to the next with its own project plan configuration.

With git you need to not use it like you did SVN....You use Tags and branches in git where SVN used Folders. I will have a Develop branch on my git repo to break apart what is available for the live site and for the development site. From there I have a post-commit hook to push all the changes to my development site then i go do some testing then if it works i merge the branch into the live production branch of my git server and it pushes to my live site.

If you need help feel free to hit me up in game or eve mail me.
Manhim
Garoun Investment Bank
Gallente Federation
#7 - 2014-03-07 20:40:32 UTC  |  Edited by: Manhim
I use Doctrine to manage most of my MySQL databases.

Using the "updateSchema" tool it can update your database structure to another one. It's also cool to make a first instance deployment easily.

So my updates chaining is like this (I'm using bitbucket and mercurial mainly, but those exemples are achievables through other VCS):

- I "hg pull; hg update" on my master branch (Which is always stable)
- I "hg archive" to another folder (removes the hg-related stuff)
- Then I edit the configuration files (They are different from production to development)
- I backup my database.
- Then I move the outdated instance's folder elsewhere (Nice to have a backup!)
- I move the updated version in it's place
- Then I run the update script (Can be automated)
- The update script updates the schema (Doctrine)
- Then it verifies the default values. If there was a change in them (Like I added a new column) it updates the row with the new values [no values are bound to a hard-coded id, this is important, they can be matched by other unique columns].
- Then it runs some additional updates like if I added a new column to a table but there needs to be data in it.

Half of this can be automated through a simple script.

This method is cool because I can run multiple instances and easily update between them. (Mostly testing instances)

- Development instance (I do my programming work with this one)
- Testing instance (I do an update like I would've on the Production instance, then run tests and make manual testings)
- Production instance

Edit: Composer is a great tool to use. Saved me lots of time managing 3rd party libraries or even my own libraries (that I put on Packagist).
Vaerah Vahrokha
Vahrokh Consulting
#8 - 2014-03-08 14:53:55 UTC
Thank you again, very much!

I think Devilen's approach is the most similar to what I intend to do.

It looks like I'll have to install git and see how it works, it's still not clear (to me) how it's meant to get files in from me and then check them out to development (and later production). But I'll learn.

Thanks also for the database tips, I'll see how much time that approach takes to be implemented.
CCP FoxFour
C C P
C C P Alliance
#9 - 2014-03-08 15:10:05 UTC
Threads like this make me feel all warm and fuzzy inside.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#10 - 2014-03-08 15:15:59 UTC
Vaerah Vahrokha wrote:
Thank you again, very much!

I think Devilen's approach is the most similar to what I intend to do.

It looks like I'll have to install git and see how it works, it's still not clear (to me) how it's meant to get files in from me and then check them out to development (and later production). But I'll learn.

Thanks also for the database tips, I'll see how much time that approach takes to be implemented.



To transfer files with Git:

Clone the repository to all three locations. (one off step)

Commit in your development environment.
Push them to your master repository
Pull them to your test environment

When you're happy, create a new tag in your master repository.
check out that tag to your live environment.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

CCP FoxFour
C C P
C C P Alliance
#11 - 2014-03-08 18:03:26 UTC
Steve Ronuken wrote:
Vaerah Vahrokha wrote:
Thank you again, very much!

I think Devilen's approach is the most similar to what I intend to do.

It looks like I'll have to install git and see how it works, it's still not clear (to me) how it's meant to get files in from me and then check them out to development (and later production). But I'll learn.

Thanks also for the database tips, I'll see how much time that approach takes to be implemented.



To transfer files with Git:

Clone the repository to all three locations. (one off step)

Commit in your development environment.
Push them to your master repository
Pull them to your test environment

When you're happy, create a new tag in your master repository.
check out that tag to your live environment.


Do you push to live manually then based on tag? This thread and discussion interests me greatly. My newest git setup is: master, development, and then a feature branch. Whenever I push to master so long as the tests pass it gets auto pushed to my live server. Development goes to my testing server. Thats me and my crazy though for personal projects.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#12 - 2014-03-08 18:07:48 UTC
CCP FoxFour wrote:
Steve Ronuken wrote:
Vaerah Vahrokha wrote:
Thank you again, very much!

I think Devilen's approach is the most similar to what I intend to do.

It looks like I'll have to install git and see how it works, it's still not clear (to me) how it's meant to get files in from me and then check them out to development (and later production). But I'll learn.

Thanks also for the database tips, I'll see how much time that approach takes to be implemented.



To transfer files with Git:

Clone the repository to all three locations. (one off step)

Commit in your development environment.
Push them to your master repository
Pull them to your test environment

When you're happy, create a new tag in your master repository.
check out that tag to your live environment.


Do you push to live manually then based on tag? This thread and discussion interests me greatly. My newest git setup is: master, development, and then a feature branch. Whenever I push to master so long as the tests pass it gets auto pushed to my live server. Development goes to my testing server. Thats me and my crazy though for personal projects.



Well, /I/ don't/ But that's because I'm bad and tend to just do my development on my live environment Blink Though for the libraries I develop, I use tags for versions. Mostly because packagist tracks the trunk for development versions, and uses tags for release versions.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Desmont McCallock
#13 - 2014-03-08 21:29:27 UTC
I would recommend bitbucket and Git as code host and VCS system. I have EVEMon on bitbucket but with Hg and we use GitHub and Git at work. Bitbucket gives you free private repo (you have to pay for that in GitHub) and Git is more flexible than Hg.
Desmont McCallock
#14 - 2014-03-08 21:33:39 UTC
CCP FoxFour wrote:
Steve Ronuken wrote:
Vaerah Vahrokha wrote:
Thank you again, very much!

I think Devilen's approach is the most similar to what I intend to do.

It looks like I'll have to install git and see how it works, it's still not clear (to me) how it's meant to get files in from me and then check them out to development (and later production). But I'll learn.

Thanks also for the database tips, I'll see how much time that approach takes to be implemented.



To transfer files with Git:

Clone the repository to all three locations. (one off step)

Commit in your development environment.
Push them to your master repository
Pull them to your test environment

When you're happy, create a new tag in your master repository.
check out that tag to your live environment.


Do you push to live manually then based on tag? This thread and discussion interests me greatly. My newest git setup is: master, development, and then a feature branch. Whenever I push to master so long as the tests pass it gets auto pushed to my live server. Development goes to my testing server. Thats me and my crazy though for personal projects.
How do you do that CCp FoxFour? via a CI (continuous integration) platform? We use TeamCity for EVEMon and have set a release process and a snapshot one.
Desmont McCallock
#15 - 2014-03-08 21:34:49 UTC  |  Edited by: Desmont McCallock
Nice double post....
CCP FoxFour
C C P
C C P Alliance
#16 - 2014-03-09 09:58:57 UTC
Desmont McCallock wrote:
I would recommend bitbucket and Git as code host and VCS system. I have EVEMon on bitbucket but with Hg and we use GitHub and Git at work. Bitbucket gives you free private repo (you have to pay for that in GitHub) and Git is more flexible than Hg.


Yea I use Bitbucket personally as well with Git, they work pretty well for me and my one man projects.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

CCP FoxFour
C C P
C C P Alliance
#17 - 2014-03-09 10:02:43 UTC
Desmont McCallock wrote:
CCP FoxFour wrote:
Steve Ronuken wrote:
Vaerah Vahrokha wrote:
Thank you again, very much!

I think Devilen's approach is the most similar to what I intend to do.

It looks like I'll have to install git and see how it works, it's still not clear (to me) how it's meant to get files in from me and then check them out to development (and later production). But I'll learn.

Thanks also for the database tips, I'll see how much time that approach takes to be implemented.



To transfer files with Git:

Clone the repository to all three locations. (one off step)

Commit in your development environment.
Push them to your master repository
Pull them to your test environment

When you're happy, create a new tag in your master repository.
check out that tag to your live environment.


Do you push to live manually then based on tag? This thread and discussion interests me greatly. My newest git setup is: master, development, and then a feature branch. Whenever I push to master so long as the tests pass it gets auto pushed to my live server. Development goes to my testing server. Thats me and my crazy though for personal projects.
How do you do that CCp FoxFour? via a CI (continuous integration) platform? We use TeamCity for EVEMon and have set a release process and a snapshot one.


Yea, I have my privately self hosted Jira/Bamboo/Confluence/Jira Agile server. Works well enough for me.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Shellac Brookdale
Cutting Edge Incorporated
RAZOR Alliance
#18 - 2014-03-09 10:17:00 UTC
Instead of creating a shared development environment on your server I highly recommend setting up a vagrant environment instead. This allows all developers to host their own development VM. With vagrant all you need to do is to create a script that will bootstrap a vanilla e.g. ubuntu image with all server components you need. Afterwards you just need to send the vagrant config to the developers to allow them replicate the dev system.
Hel O'Ween
Men On A Mission
#19 - 2014-03-09 21:06:57 UTC
Steve Ronuken wrote:

To transfer files with Git:

Clone the repository to all three locations. (one off step)

Commit in your development environment.
Push them to your master repository
Pull them to your test environment

When you're happy, create a new tag in your master repository.
check out that tag to your live environment.


Well, that's some useful info there. I just started using Git and was wandering of how to set up my workflow.

EVEWalletAware - an offline wallet manager.

Pew Terror
Sebiestor Tribe
Minmatar Republic
#20 - 2014-03-09 21:35:44 UTC
It is hard to make suggestions with the given parameters.
There already have been a couple of generic suggestions (tagging/branching vs CI environments vs isolation containers (docker/vagrant)).
It pretty much depends what you mean by "big" (as in data or processing power or bandwidth).

Personally i would also stay away from PHP and start off with scalable technology (like node.js or azure).
123Next page