hello guys is under very and grady leading the announces admin team and this for very is correctly i notice a sub men will be telling you about how we make up twenty min age okay so a bit will be telling you about how we men age our influx store out of site softer side what will be the future plans of the infrastructure a lot of more stuff so i live at the word to three for the first just lights okay so what we do so we met can so seventeen so we have a number of roles we have been making infrastructure so that means number web services own about things like that of course we have get those kind of things our main job to make sure everything is stable and you know and running smoothly right so and we also try to follow but every single team in the rubber in and copyright is to have a problem so it something goes down or we hear about is the right we are jobs trying to fix it we also coordinated our hardware sponsors for anything related you know for is red hat we work with red hat and so forth a big ever was our sponsors i would doubt that about the next slide well our sponsors are currently read a canonical down they are gonna state university the open source lap so do the right down in fact sort or has been really helpful with us many times and i'm really thankful to them and you can switch the so i would do we communicate with each other we teach sys admin we do have a channel on that your see that's going on that are which is sys admin we do have a public mailing list and we all set up private one where we receive all the programs all the proper grounds domain renewals also we discuss and it in private like private matters and we also set up a lot of new services for better can users and developers to know exactly what's happening when i went for accept or and these services are start does not than on that are and we have a we have a little then we are that's three will we show you okay so we have a good bunch of services and indicators something is wrong you connect to the service or you can even keep this page on your are that's filtered and you get each update when something goes wrong so let's say that something's wrong with other but we sure are red that buttons saying that there is a problem and we have quality checking what's going on there so keep this page bookmark every time and before asking us on the sys admin channel get yourself on this package and you know what we don't really know what's going on and okay then we created i'm not remaining liz i'm sorry for that but when i'm sending my announcement made as i was sending them to desktop double is a foundational list for mission announce so they created in fact that are announce whether i'd be sending well my emails about what's going on only for accept or and i also created a public not just you if you connect to ideas but the number per week to username anonymous and plus or anonymous you see old the services we run and the status of them so it's even better than the start of the dog the them that or one because not yours is a real time service so it sends a checks to the machine if the czech was wrong it repulsed that if i there immediately so just i don't have a preview of that but you can check that on your or your machine stuff okay so let's talk about the artwork read a is kindly austin a lot of machine for us a total of twenty two machines both vector machine and physical machines at the phoenix on their things chewed at the centre then canonical exhausting well two machines none of them exhausting the guy that website and they're going on website and down the oregon open-source lot exhausting i wanna not be as box and the testing machines for that website themed that and vested you about on in stock we do not see lee use lee peered and capable and we actually have just eight physical machines which are but huge in terms of a around C P U and so we use a lot of we promotions and there is something special and you now we name our servers for example demolition that currently all's a going on that are it's called socket many other caulk on the box or create bar or something like that and the name is are taken for day G T K we just so if you will do a reverse D N S check on a rolls you'll find out some strange names don't worry because is that is that actually expected okay so let's go to the software i will in age our web servers we do have an apache server in front of H a proxy so we have should bucks as which acts L's front ends and the fact that can machines actually are E P less so they don't have a public E P and there couldn't you just accessible by this is something team and i'll does H a proxy works there is round probably in the innocent the N S entry so you're clean and will randomly get an E P and connect to one of the two products is and this is call fail over so i wrote boxes are set up enough a little remote and we use an apache in front of a chip proxy because we're ready back well the best to ask ourselves so if you to check well than on the non services well down are us which ssl right now and of a can machine are abused lee adjustable or only by a lot that can also something so time for free okay so on the software side we have a number services or actually more accurate way i'm kind of i'm going to describe what environment looks like so i'm not so machines everything is N F S mounted so users generally have a consisted you still are going well not users but systems as ministers on the whole network we have open P N for we she's posted outside the phoenix data centre for actually the private network so you actually have to dpmc axis these then the phoenix data centre and that's just right for security purposes we also have open L that for our managing central access over the good this is primarily how we do accounts i think it's not in those people know what that is finally we have or are buying for serving guinness i don't think there's much to say about that we have to name servers well to cashed us requests and then finally we have a gonna jabber server which book talk further and slides the next that we have this prospects and the spend virus cover to so jonas our mail server and of course we have some stuff to catch spam we germans were classes web platform for about that be that the minimum that gonna that word and black second another probably can sample are using more annoying and that's a and we also have okay nice extensions that can and other which is so i per class with an that was about but the by jasper and then we have you can that which is a nice note taking application that let's you it's text and data that whatever we been using extensively at the marketing team especially and using it extensively for taking notes for on like prayer meetings like honesty and so forth and now i've got it yes so i we manage ourselves we evenly use pop but we do have an internal proper repository manage we the get and each comment we do is than notified to all this is augmenting so i actually know was working on wall and all the team is aware that they shouldn't archer specific model because i'm doing some changes there i'll pop i'm not sure i know many of you blocks leno our puppet work no one so top it is a centralised management system so you keep your configuration files in one place and that one place is called puppet master you down synchronise you create model models and you synchronise your machines to a specific model so you are able to recreate an entire machine we just two clicks and by running a specific common so it's really and they to rebuild an entire virtual machine in the case for example you're operating from well five to wreck six we also keep are standard operating procedures on our we key because each softer women age is configured is i evilly customized so not this is something is are aware about our we did a custom on is that we key or like as a lot or adapt so this all is therefore do specific motivation you check the soap and you have all distractions you need to find out are you should probably men age a specific service that was set up by someone else so we come to the backups are you we don't have a back lit a machine and we manage our backups in a strange way i mean we were on a nightly us think around that sense well the doctor of all the service to a north carolina know when to an or carolina facility and from there we get all the backups on the tape on tape drives and when vote a little script for that i was going to ask and exclude list that means that we can exclude specific directories from that we of the for specifics five system and exclude i dunno cache files for example or flies or been mounts so we don't duplicate the data on the on the backup well the lots of us sink are then sent to a machine well and that machine generates i'm apart every day so we get the mail which does something like well the machines are fine all the backups one fine or something is wrong with the of with a specific machine so you should check what's going and this is called a tear apart and you should think we should dangling providing just very nice by transcript so i want to we do our monitoring we are easily using now i guess using not just with a lot of customisation and the main one is called czech and K second czech and K is a no additional software you can actually willing to not guess and you get and ten or more of more checks without doing anything special it does that for it just work out the box you just up to link it on when you're not just interface and configure a few things we also have i don't and those are really nice feature because sometimes apache can go down and for whatever reason or you just got that time out i don't and those are there for that and our that are those they work in fact they send a check to specific service and if the czech is critical it automatically or starts they H T P D demon so they are very and a and then we have P N P for now guess which is a C E mail are too i don't know to comedy or to collect D it creates a pair often service statistics over the time so we know a real i'm not then machines slowly is or how much ram is being used during the during that time so the future plans which answer many of your are interesting and many of you are complaining about bags a lot because our but still a sense is very old is version three don't for one of the neural is far but for but why that's great didn't come yet the normal as many extensions customise we customise of our but still a sense with many extensions that the we have written by a lot of readers because i don't know if some of you know marine and unfortunately is not contributing anymore and why talking with william the other day you offered in south to as many or of writing these extensions to the new but still a format so we can probably planned an upgrade of our but still are doing the next six months i guess it will probably take that we will probably have to take down the service for around two days i guess because the them the database you which is around twenty gigabytes and restoring the world down will take i wanted to six to forty eight hours i guess the next future plans is or a locating are machines at phoenix to currently our an actress so you switch the networks which we are currently using is the one of another rack and that's definitely something we have to fix and we are collaborating with the field are guys to phoenix to to have all the management consoles attached to the breach of the machines and so we can track down well the errors of the console of the physical machines our doesn't work we thought each machine also management console part you connect D is mush this us which this part to the natural speech you get then E P i username and a password then you got yourself into the V P N and you cannot assess their web interface that allows you to report the mission check the console rest are gonna should down the machine on song and get all the data else about the method man a vector of for example so we got all the details about the physical machine itself we also plan to remove shots from master because going to there are a lot or a lot of active accounts and well of them do have S H adjust to master and i filled is that this is not very good secreted speaking so will be removing assess to master and we'll be creating a script to do that for your locomotion and we have think about quitting hourly the web interface what you cannot indicate using your old up a sort and you can just then click on upload file you select your top are your trouble from your pc and you click on in style this we call death dp admin script from the machine and do the job for you so we finally got rid of of the shelves and really proud of the last season and thanks to about through is what did all the work and will be monitoring the work it not the norm to talk to get out and this without plan program i think applause festival of so when you comment to the to the repository it to we run a little nearer that will saying well the content of your of your model you have to get up but will be announcing that is in a better way with more get those in the near future i guess which will the instructions you may need it's just are coming or it's okay you know alright and the you want advantage is that we can track the activity around the particular project we just one click in we can see who's working what they're doing against master right so resistors and i i'm certain your get more details later cool well also given the fact that katie the are pretty bad this group shown i don't know a few months ago and the they didn't have the backups so we are also mirroring to make sure that we have to backups one sound to the to the external facility every night and one get up we did to time their guitar people because they've been really up for we does and open a about or setting design so actually okay so this part is really the fun part this is the part where we get tell you what you can use an and to help you to have cooperation the first one is me but which is a it lets you do meetings on I R C and what it nicely it it's a it's a software that most people don't know about which is why i'm discussing it here we got that's you if you want you want to start you just say like you tell be bopping gonna start the meeting and then you are able to have all your discussions lark you can actually have somebody has a leader so for it's really quite cool and mostly done you can just kinda and the meeting and then you automatically get lots published a on a web on the website now i don't have some complaints about this because a lot of people want the logs to be private and so that's what the feedback we've gotten and we are gonna look at how to do be lot on a per channel basis and so if we can be configured like say if you sort of meeting on like how you know right you can expect that's gonna be public whereas if we started one in marketing or you know some other channel then those would be private so that's one thing there D this light has a link to be documentation so i will encourage you to look through that use it and give us some feedback because i think it'll really help us out when trying to collaboration the next one is you think that a talk about this before we use is a lot and especially on the marketing team we did this constantly internet quite like assembly said we for princes we were putting in but we're doing summarise so each day and we've been using this it's i basically you can have three four people adding this this page and you can have chat and you also know who's type being white because a lot different colours it right we can always try to do a demo real quick we've had some feedback on this as well you know the one of the things that happen on this is that you only get one either bad and if you have any us either pad before a lot of people would like to use multiple because when you go to it you only get this one page and if somebody's already using it then you don't really know what to do it so you end up people and of going to pilot patterson last so we're looking at trying to create new ones on the fly so we don't do it before because we're worried about spam but we will try to doing experiments if it works and if you don't get spam right if we don't get spam then well we'll figure out what to do at that point and we had a whole session on this well but if you haven't been to if you didn't go to that chuck this is a this is really cool because this allows you to do document storage right so we document storage you can drag and drop whatever like open up this or any number text sometimes if you have graphics design things like that it's a nice place to put it without having to use google doctor some other or other some other proprietary stuff so the other cool thing is you can share that if you click on it and say share with you can share with everyone you could create a public url just like you can't are dark box and it's it works very well so that's another nice thing if you wanna use it to create about and we'll be happy to give you access has using all that much of the secure that what about like to encourage them to use it more so are you okay well then right need you put some screenshot so something like work in progress markup of say website or whatever or just like a simple stupid disposable thing right which store some services for but they are like i know how they are right so what it used to do is to run this in a temporary folder on my own server a but i'm very happy that it's a this thing excess because it also i lost thank you very quickly easily charing public files winners and yes this is what is looks like sorry so thank you for sending data and the guy course is relevant work an integration with you know right we'd gone online docking online services i want accounts are some reason i remember but reverie doing arrived are putting like it's gonna show up are gonna documents and so for so i was just wondering when it is you just going to be for national number for like the Y O the foundation members only yet so this is sort of a something that you get for our as a as a as a benefit so we should we should definitely do that i do understand that it's probably us too much from that if structure just to make it available to everybody but is i mean it's a service we expect you that you're gonna use it four gram related activities as opposed to you know if it was why then you know we would end up becoming a service provider right if it becomes too popular we might you might have to do something else because all that stuff takes place and you never know the on the other it is to maybe have a monthly payment yes that would be a good a good a good go to okay not the the main concern i have is about privacy because opening are instance to the world we sure provide some of the old documents saying our paul is our legal places the S so okay so i will like to take a few minutes to remember frown a great friend i work at we sat for mass i can tell to us actually on the for the way facts or and it was and i mean hazing and amazing better sound and you was you always helped me in many ways you was very busy hiking on many projects but every time i had a question it drop by don't question or a sys admin question you was always available twelve and i will never forget in and i think i wanted to create a slight so we can remember em hold together or okay the should be able for me if you have any question it's just right well the thing is i if you have services that you do want you know big even discussing here it'd be nice i does one person wanted L X R four could searching so that's one one avenue there do you guys have other things that you think would be beneficial gonna a building windows and mac binary is on what exactly it stars or i guess i don't know i want i i'm here you can king use the use i was thinking about motion maintainers and that there's been some discussion about doing build windows binaries on the from build like that or we have a seem to set up to do the old will be used for a always project them or so i mean i think it's mostly question having somebody to maintain build scripts and that's for the across to automatically why do native bills have windows be and that's a trick your if you something would have to work all the are we are how to properly but it's only doing crosstalk with like a menu double you is certainly very feasible to really question for mac i'm mac is are difficult i think we'd have to have your some way of getting back centre infrastructure i think that it's it is these it is anything is feasible what it's a little bit more tricky so about a well i'm sorry what's that exactly you don't know victorious okay so i i'm sorry i don't actually we so people will only well if we're doing you have clone you don't know what it is exactly is the reason is the we closed are using pull requests we don't wanna do the requests and intimate i would looks you right just we don't wanna do that with the close or service that is one team and someone else but we like internally to be you really useful for contributions that definitely you to put it up but it will take a look at it so there is the that is in this but five back in two thousand nine problem at the comments and analyse it well there's like this to submit the transmission file and last number one and we'll come in the final after a you hang on that is that the main problem i see in setting up a guitar is first it's really our to stop really and manning sys admins told me that it's being an item our well so are we going away from our work grew around i mean we do review patches and sailor and will be spending a force in a pretty bond sale or does it make sense to you setting up on order it's tunnels which is it orders i mean about the per class so if it's not good have the projects i don't remember the name of you know what is that right i don't know how to discipline a because we really happy with it well the problem is that gets allowed i don't or is introduce a completely different way of making at else right and well if it's not feasible then we don't that no women is that women H everything propelled all and all the scripts we use our freedom to talk about work tools problem held up instance right so that mean providing all the scripts for another or another service like it's about which is completely different H else system right so it's not something that can we do if it's not really of them we better users as of or something else i guess the last to the other question i from my experience a reviewing pull requests this a lot use here and then i would have to you are the you percent that in something like could have or ataris is i guess it the they work flow is a lot more it's nice you can also i mean we really we have meter will be able to do like we compare comparing that the master runs with the future ranch but of course will relying on a on a also service so it's like we i don't think it's nice to have the me that i don't think that we shoe we want to ask people to use a aggressive so receive if we want to the things nicely so i don't know but maybe we could do is fill out a way to just percent that they have without setting up the whole infrastructure there was also proposal to so top review to review board but again sure is just a review board which is another review tell and then we have splinter which is you've used by but still and other non developers so that works role we got tired of i mean you send your patch a review board and you say some top out there you say that but the thing is that right now if you're working on a ranch it's already if you're using was that there have to generate the patches and then i will a hole in have some information with language and able to that but still a about so once that not having another one at the moment our but still a sense is completely separated from there is no interaction between it so you have to my only great departure submitted to but still and that's all we nice about and integration system between but still and i work at but that's also a bit harder because we use to get which is us off to return see that does our web interface craze this also on the goal of something like that that's we have some have to live in see right if we were a if someone wrote be able to expansion to do that maybe we can great on extension for but still okay want to propose i'm not suggesting the you do it up and then you can asks if someone's up to do it so we need more manpower wide come to develop new tools right now i just suggestion to the A B and they have to are well companies organiser it is nice to have a open source event manager to centralise to happen environment to or scalable one and the problems and but you but the troubles will probably you know this is more professional for programmer number us to be also well all you have if you're review paper submission and then out emails that they're too good during the process so that's how project if you guys have any suggestions any ideas for this you should look for disk are just too it was is a dog maybe you got you very tall because i i'm sure specific a reference the things you need for no way show right so i think that can we know it is definitely welcome you also should scroll to building or management system for conferences so just a lot of time what i know workshops it's not question i just really wanted to think you guys you've been doing and a really awesome job be i mean especially for collecting it he's been really helpful and i reassure all the other team's has the same just say sent thank you very much but