alright so

i'll just get started and what people you felling in if you come so i'm

gonna be talking about this project called know mostly it's sort of an awkward name

but donna names not too important

before i get started here though before talk about what i think it's really important

to talk about why am here why i care about what i do and the

most important thing that really drives me is free software very passionate about it and

i just think it's

it's always been important sense ever since

which is a men road

started the G P L and before that secondarily i just have a lot of

fun working in this community and

you know hang out with other people in interacting with before the world it's also

but fourthly right having a prize linux is very important to me and know was

a very important component of that had enterprise linux

and so when i ship

this product to our enterprise customers i wanna know that known is providing red hat

enterprise linux with high quality software

so all these things are important all these things drive me

so secondarily though it's important is to like restate or assumptions briefly why assume here

well this is all from the website obviously the free software the software freedom part

is in common with michael's the other ones are important

but i wanna dive into this connected aspect "'cause" i think that's really interesting part

it's probably the most interesting to me so this is sort of the way i

the way i think of it is gnome is in ecosystem and we have our

centres of gravity i'll talk about those and then obviously we interact with all different

kinds of other ecosystems so greater or lesser degree looking at this list

you know obviously we share a lot with the canoe project tries in is there

an use this on components right enterprise linux very important to me obviously uses a

lot of know and there is there's

the programming languages icons for roles monos

and other ones like open in embedded which in turn are linked other ecosystems you

know open about is going to wind river and all these other embedded devices and

stuff like that

so

it's important is sort of just visualise

there's all these ecosystems around the gnome is there around the centre grab these that's

the way i think of it

so

with that background what a minute talk about is just to build system

but i'm really excited about it which you guys made notice if you see it

online

because i just kind of always wanted to send it's the in so successful now

so

my goal with this bill system is to focus on owns core or centres of

gravity and

not exclude other things but just make sure that we're doing well what is important

to us and all talk about that and the continuous aspect so it is building

but it's also testing continuously and i'm i could talk for the rest of this

whole session about continuous integration why it's so important but i'm just gonna gloss over

that

so

it's testing

a lot of get repositories a lot of the what makes up cornell so kind

of when i started this project i start from this is sort of minimal system

and build up

from scratch and so i got to know him shell like i wanted to just

minimise interactions with other things so we can again focus on our core aspects so

striking a lot of get repositories for example all hundred and some X or get

repositories most of which are basically just the protocols and never change but

and of course all the know network the think that are important network manager system

do i wanna make sure that known works well with the latest system so tracking

system do you get master very important

the basis done this is sort of stuff like that i just don't care about

like for example an S P R and S open ssl those all come from

of an embedded and basically are ran make some binaries popped out if you're to

work that's about the level of Q A that i gave to the

also crucial is that i have

a sort of course focus with this minutes only targeting virtual eyes environments because

there's many reasons for this one is that doing testing fertilisation is whole much easier

than trying to have like a test lab wired to physical hardware that's a whole

other level of management personalisation it's really easy i can script using you make it

do a dance when i wanted to it's great and of course

there's other aspects of this like there's no security updates because you basically need people

there employed and obviously my employer does this

i'm not trying to compete with them obviously so these are that's sort of high

level summary of what this project is and kind of hopefully just is important what

it is not

so the core values of known as i see them obviously we have the G

T K stacked you live is very important to me because i think it

it's a fundamental pieces for writing modern applications it has the event do it has

a file system abstraction a bunch of stuff that really is a core part of

an operating system and i'd like to see it used more particular in red hat

enterprise linux

also in known we have accessibility internationalisation also part of our core values and he's

the secondary list is kind of my list

software engineering very important to me get the all the basics of doing continuous integration

you know just are we leaking memory you know just

again the basics and i'm trying to sort of push for

split between the operating system applications and then always exceptions and i'll talk about this

a bit more later but just to repeat again the goal the system is to

focus on gnomes core values and feedback code continued a feedback as we have been

doing for years back into these other you can systems

so

the non goals no enterprise features no packaging know enterprise features there's no you know

if you want her breast for example that's not something that you know i'm gonna

pass

and a fast home again not something about it

testing G T K G but

so what has a system achieved well it's been online deployed in the known data

centre since december ish and since then it's been building fairly reliably and very importantly

without human intervention i'll talk a little bit about the details of how that works

but a lot of times it's just building one off

doing something else and it's just working fine

but more importantly than the building it's been doing testing and all of talk a

little bit about the details of that

i have some numbers later so this is something that i'm most proud of is

so but i'm from a get pushed or something like do you live in so

when i'm booting a virtual machine is down to minutes now and

it depending on the type of change

like

depending on the type of change i think i could get that down the second

so that's always been michael

it's totally possible the other thing that's really important to me and i talked about

this last year a bit but

the story for how upgrades works is just crucial for no and

something i just find deeply embarrassing about the current red hat enterprise linux six which

i run is that the upgrade system just doesn't know anything about the browser in

just it's not safe to upgrade it like you know if you're if you lose

power halfway so part of this project has actually been redesigning how it this work

and so using it was three system upgrades actually fully atomic you can pull the

power well they're running and it's just sort of crucial

so even more interesting there's a lot of tests that right now so before developers

kind of ran make check

periodically manually now have automated test automated make checks of a lot of different models

there's

three hundred fifty test binaries which in turn because of G test are in turn

out a lot depends how you measure but this takes a couple minutes all talk

about that later another cool thing is that for a long time unknown would really

test G D M that was kind of ray would run makes you will make

install in his laptop but it's not something that fit the J job model right

J scroll only affects your session and so sometimes G D M well it was

kind of broken but now if G D M breaks all know in minutes ands

that's been working like

so i want talk a little bit about the always three system and how its

kind of different from the packaging model because it's

pretty it's pretty crucial to how the system works

so i buster itself is just kind of like a user space filesystem enter and

i like the term get for operating system binaries

it's misleading in some ways but on the ways very accurate so what industry does

is it manages to let you have let's you more parallel blue multiple operating so

multiple versions of multiple operating systems

without affecting the physical root directory of your operating system

so

it's like get in that it's a content address system so it naturally to duplicate

so if you have multi same multiple versions of gnome installed they only use space

a by proportion how many files are different

so

i'll talk a little bit more about that later to it's different though from a

lot of image based appointed systems that happen prototype out there like red hat enterprise

linux have this state was project for a while which was basically a kind of

image of a system where you download new data to block level and you're at

sea is N F S mounted or something like that all those trees different from

that because it does operate at the same level as package mentors do but unlike

package managers it just a done content replication system in this allows a lot of

neat tricks i'm a show those the next slide but straight the way to think

of is really am your image of packages are lots of little filesystems with some

shell scripts that run as root attached in their upload in the internet and you

assemble them on your client machine string model you replicate filesystems from a build server

or you can generate them locally

and it is filesystem independent so it does leverage some butterflies features mostly through G

live you we automatically use the better fess i aukro for a copy and right

type stuff but i think it's sort of important for general purpose system to work

on top of plane yet see for and that's actually what i test on the

most because fifty fours really fast

so

it does have a format for installing on your local system but it also has

an archive format they can run on a web server and that's sort of how

this the build system works as there is than almost resides the build system it

commits to noticed repository and then you can download it directly from just a plain

old apache

so here's some tricks that i wanna make possible and i've been protecting this a

little bit with laura is so i run redheaded present X six on this laptop

because people favour subscriptions any bill fix it but it's obviously important from able need

to be able to run known get master because i wanna be able to contribute

to it and make sure that it still works for the next version so

one the way i plan to have this work is all just have reading representing

six am i root partition and all street just sort of installs inside that without

disrupting it so i can have the know most re what's called build master just

the thing is tracking to get repositories but you could also parallel install multiple versions

of or a very easily

the hard part about this is fixing or P M to not be the lowest

level anymore so that's still in progress but i do have the point where you

can have the base system and then to the newer one and test it working

fine

and when i say five hundred builds here that's really possible because it just comes

down to how many files are different and between

things just don't change that much and there aren't a lot of big binaries so

you really can have five hundred builds installed in more interest a need or trick

that you could do something like automated bisection which is where you know you have

your starting point you say okay i get graphics corruption recently and sort of write

a test binary that maybe just takes a screenshot what for the corruption and then

just keep booting into all these bills and so you find the build it's broken

and from there you can from the source code i really need poster trick in

this is kind of part of how the cork you a system works as it

can up update virtual machines while they're entirely offline

and this works basically because it's a don filesystem replication and not like packages where

you have these ritual scripts attached and what they do

was she has none of that and so that's this is actually a very useful

trick in general and like i mentioned before i'd like to see austria eventually slide

underneath these package systems like D package in our P M and all that stuff

where or she would give the parallel booting the atomic upgrades that sort of stuff

to them

so know much three is layered on top of austria and it's very simple there

when i say there are no packages i mean i went through the entire system

in eliminated the concept at every level as much as i could to try and

gain reliability and sort of optimised for

automation

so than almost relay or has this thing called a manifest which is literally just

a list of get repositories to build and work

it's very simple and as a consequence of that simplicity it has turned out to

be that has six months of and very reliable promote stories and so if we

look at like example of what the manifest looks like well that's multi one that's

the entirety of what would be a package in my model it's just it's just

to get repository insert it'll to build in this order and that's it that just

means track it master of lib X M L to

and spend enough for quite a while

the component about one i'll talk about this on the next slide but basically that

means this is only for the developer image i can also choose any get revision

to build it anytime unlikely package in our can none of this cares about version

numbers at all so i can be tracking system to get master and say okay

i can no longer log in and the current version which has happened so i

say okay all just tag back to the previous revision builds done like within minutes

it's completely on like the package systems because they have this deep obsession with version

numbers that

permeates every single layer so none of my system cares about version numbers all any

get revision at anytime

and is the spice example is just how configure options and patches work i originally

was trying to only have get repositories and no patches just to force myself to

feed changes back but i sort it's too impractical so i ended up having a

couple patches

so this is something i think really illustrates just how different it was tree model

is from the package thing so on the build server now it's been doing continuous

integration since december right there is over six thousand complete filesystem trees like i haven't

lost any history right so

in the only occupies sixty gigabytes and this is sort of the power T duplication

so in theory again not only can you have five hundred bills locally you could

download in entire history of all the bills from the build server and

you know we'll get those all have links to their sources inside of them

so

i just give a lot of flexibility right you can have a sort of a

system and then add remove stuff we're supposed trees all about the files so the

way all history is sort of designed is that sort of having a bunch little

packages you have full complete systems that share space and i would you have multiple

versions of say what i call runtime tree but you have a basic runtime tree

enough to run out and then i have an image div L with all the

developer tools and these just naturally share storage so us three let's you switch between

atomic so i can i normally run the developer imagine my D Ms but you

know sometimes we've had boards where things because of obscure issues only work in the

developer image like a they in package world they needed deborah develop package installed i

can easily just switch back to the runtime tree which effectively hides all the developer

tools so

parallel installing multiple file systems and then besides all the for variance is actually three

it's a the build system outputs a pipeline so is what i call build master

which is all these get ready all these get repositories get bill integrating the build

master all that happens there is there just build a just take the latest get

run make install duster and it's in the tree you can download that that's very

useful obviously but

i don't expect many humans to want to run that directly so there's a secondary

stage which does what i call smoke test and this just updated disk image puts

it union does this appear to let you log if it does that gets attacked

just like get repositories get tagged

and there's a secondary stage which is integrated is that after all the make czech

and stuff like that works but

the way i sort of vision this is there's a like a long pipeline you

can imagine of the very top we have build master and then something like red

hat enterprise linux is like way a down there but it is a sort of

long pipeline of extended increasingly extensive testing

but it still really important to be able to run the build master because sometimes

things a broken and i wanna be able to debug it locally so you can

download develop about run gtd on the latest G D M and see what's going

on

and there's also at a task the generates disk images

after something

so on talk about the story of

after you run get push to run in to get repositories that are tracked your

what happens

well as far as i know on the first person ever demand push notification from

don't get so that happens and then we basically do a copy of the get

repository under the build server

so this bill task though important thing to understand is be in a feedback is

really important to me versus correctness their different tradeoffs you can make your so the

bill test only rebuild get repository that change because concretely i can't rebuild look at

you take a every time G T K changes it would just massively slow down

the cycle

it does

it assembles a build route which means it's a clean build in this is actually

really important because some things just

it just really increases reliability but this part actually really fast compared to the D

package people there and the mocking those sort of tools it averages anywhere from five

to twenty seconds and that's because it uses hard links stuff actually building the thing

it starts from a clean get repository

so this is also kind of important because it'll for example instantly catch a missing

file that you forgot

but for something like to live that's a couple minutes and that's the same with

change builder can whatever and then the result of that install gets stored into repository

and

this is kind of like the package that is

this us the string here than almost three components to of X eighty six sixty

four that's kind of like a again branch big it's torn repository and then we

assemble all of these it just takes a union of all the different components so

i have to live i have lived to stored has their own little filesystem trees

and i just layer them on top

and that's very simple so there's no dependency resolution there's no packaging here and that

takes a minute right now that's something i could speed up but there's different trade

offs there and after this again this is just like

for and i know change of for a very small module this is around two

minutes from get pushed to when you can actually download it reliably

there used to be the concept is yes is the short answer and i used

to try and you triggers on the client side like the idea was that you

download this tree and then afterwards we would run stuff like T update i can

cache on the client

but it's i moved away from that because

then i couldn't do the offline updates on of the M and that was really

crucial so i sort of took the hit of having each multiple versions of the

cached icon caching each tree

it's not that bad and it makes up it's a lot faster

baseball for the client this is one of the things it does slow down the

build process for example on for some things i have to rebuild in a drama

fast whereas at least for their and i don't know about that the end rebuilding

the trim F S on the client every time you crouching so in some cases

i pay a price on the build server where is in the package will you

pay and it's client

that again sorry

how many different triggers

seven or something maybe like for hang go free type

i don't know there's not that many L D config

i mean i can look really but those are pretty fast execute

so

this i'm not excited about just building again i think the passing parts really important

to me so use live guess the vast amounts the filesystems non root

so the smoke test the way it works is we have a cash disk image

representing the runtime tree you know without all the developer tools and then i make

a little copy on write disk and i had some testing stuff for example there's

this service that runs in the virtual guest export the system you journal back up

to the host which then looks for stuff so system he has this nice concept

of message ideas which are

basically Q I Ds universal unique ideas that some event happened right

so an example would be system D M it's a message with a particular message

id want something court

so if someone makes a commit to say network manager that "'cause" it just simply

say for startup

the smoke test will fail

for them are patched gnome session timit a particular message id of one of the

things get manages fails like so for example if gnome shell exits with an air

or i know that pretty much instantly on the hose and i can terminate the

V M and then if you see nair see the gnome hackers are see a

red line for the smoke test that's what this

so most often it happens when people commit syntax errors in the norm shell javascript

files

lately anyways and i detect that very quickly

and the integration tasks this is something that i think is really need to which

is before the gnome release process each operand make czech kind of periodically

randomly

and then theoretically you ran make just check before release so in the know most

remodel i have all the tests almost all the tests that you run and make

czech installed in the system so i go to virtual machine and then i have

a little testing framework that runs all of the tests after nes change so it

change translation update an am show i will rerun all that you a test i

will remind all the clutter test because honestly have plenty of C P U power

and

why not i mean it actually i wanna have

you know a more interesting test scheduling type system but

the build server has thirty two cores sixty four gigs of ram in flies of

the stuff

so what that is occupied so my time for one hours race conditions on the

tests

do you live in particular has a lot of tests with race conditions and

that's

some of those a bit hard to track but fortunately most of them are just

bought some the tests and not actual core code

but i didn't i did find one conceptual issue and you live from tracking down

these

but for the most part most the test did work already because the developers we're

running just not relied continuously

so just very different

so some things that i'd love to do with the system kinda have

twenty minutes okay

okay cool so

open sousa has this really well they have a this thing called open here which

has some really cool features and some kind of weaker features but they have a

really nice but you and i love to have something like that another feature that

i wanna add is mostly has try server so i wanna have does

does this patch break known as a service basically where

i mean there's no reason i couldn't just start building inbound patches and bugs a

lot and testing on so you know one an optional developers can upload

but

bunch of patches and i just take those do a smoke test that sort of

thing would honestly be pretty fast is just the tricky part is sharing stuff between

the different builds like the get all the get repositories are out four gigabytes and

i don't wanna read clone them for each bill but making sure like and linking

currency is just what's kind of starting a little bit from this

and i would love to get into like software engineering stuff and one thing that

i've always wanted as you know just a graph of how much memory were using

on the default login how much memory using one epiphany is and wikipedia i mean

just

also the data there's no reason i can gather this the virtual machine aspect does

the limit kind of the type of performance metrics that i can get like i

can really get approach are because there's all this P N I or activity in

the build server and it would just be too noisy you really need a nice

dedicated harbour lab

probably

that's true

i think

i mean there's so many different ways to do it i don't know if you

knew has the hoax i would need for this i mean that would be a

pretty reliable way to do it i installing agents in the guest is obviously one

way together a lot of stuff but us

then of course you have the heisenberg effect where you you're affecting what you're measuring

right like your agent is using memory your agent

so

i would love to have automated bisection on the build server so you know right

now is if the smoke test fails i basically just run get log and

you know it's usually pretty obvious

who broke it and why but a lot of times it's not actually and just

like a try server there's no reason i couldn't just allocate a builder and say

okay i have this regression range well the know most remodel it's of course over

two hundred get repositories so

handling the multi repository thing with bisection is an interesting problem but again there's no

reason i couldn't do it just haven't gotten around to it

another cool thing is a little bit more blues guys just a sort of live

collaboration setup where the developers actually running get pushed to a branch and we instantly

build it so you have a some work and am shell and it designer and

that you know develop design i want to see the developers doing there's no reason

i can just okay to server that just sort of tried to do that as

fast as possible let them almost live collaborate

so that's

that is not shows the system but i have a lot to talk about where

i see the high level future the system so

obviously

okay i guess i didn't mention this but there is no application installation mechanism in

the system yet because it's a hard problem it's

very hard problem and there's a lot of prior are in that sort of stuff

there's like most all or have re pronounce it there's a bunch is click packaging

is all package on the stuff around and they have different tradeoffs parts

the thing that the thing that

is very important in this discussion i think is to draw a very strong line

it's very strong distinction between free software applications proprietary ones because i think that demand

there's some commonality there but

i don't understand why so many people in free software have this obsession with generating

binaries in caching i mean the most important thing if you're talking about free software

is the source right and who's committing to that and why and not this package

stuff so for free software applications i'd really like to emphasise that sort of stuff

like who's writing this out you know let you let you run the latest version

help you contribute to it that sort of thing

there is the aspect where

it is important to say who built it because you do yes you obviously have

to generate binaries to run it but it's still important to say who built it

so for example if you know if we have a system for applications we're doing

continuous innovation and fifteen outputting zip files or something who knows doesn't matter it's still

important say build by know because you have to trust the builder

and so i think i think the aspect of the intermediary is still important

again i would love to do continuous integration for the applications i sort of paved

and added some core apps to the manifest so you know obviously epiphany is in

their gnome terminal the system would be completely useless without it

and those are kind of

those are available but the really tried you wanna get to is where you can

remove the web browser from the operating system

apparently that's important and just you know peer review i mean this is the same

thing i'd like to see for more for the core operating system is just emphasising

the peer aspect of free software

you know again who who's doing this and why

okay

so obviously this is gonna kind of controversial topic throughout norman it really gets to

the heart of the distinction between what an operating system is and what it distribution

is and that sort of thing which is

where do you draw the line for what lives where right what might applications have

to bundle

so stuff like the backchannel to i'm not a big fan of the back small

to i really wish that it'd been but you will nice to a library before

so i

i you know do we except that is barbora base platform and it's ugly api

i don't know stuff like hire O obviously we made the decision

this is par the platform it's part of G T K spell checking is pretty

much perfect example of something where well we have some libraries out there may not

be great

whether i could should with the base system or not i don't know

and of course even if we're just talking about court know how long do we

should you take a to do we encourage do we allow

allows only should application developers still target in some circumstances and of course we're talking

to do you for enterprise and how long would we should G T K three

well so the way i define operating system is something your application will run on

for ever

and this

if we remove stuff then we're not really an operating system anymore kind of moving

back towards a distribution model so i don't have answers to these questions

a sort of punch on it and said okay well i'm just going to

to you know to make an operating system we need to have basic quality assurance

and so that's what i've been working on for a while like just fixing race

conditions and the G let us and make sure making sure that worshipping platform the

can be used to build applications and how all that works

so a lot of questions and so there this

so i think proprietary out the needs a proprietary applications are totally different

and

i mean

for one thing they

their time to an architecture usually if we're talking about me to binaries and all

of a sudden you're kind of in the windows world where you have

applications that were targeting X eighty six and then if we wanna move to new

hardware platform we can do it

i think that sort of thing are few icgs can just sort of deal with

it so i'm not too focused again on the proprietary applications basically what i want

i think it's important to enable people to run proprietary software

if but not encourage necessarily and so

i'd i'm gonna focus my time and just the core free software gnome applications because

that's again why i'm here from again that's all

so this is something that i was actually hacking on the plane

and which is basically just system D for the user session this is kind of

just a for corporate system feature but

using this we can actually this would be something that would help enable running proprietary

applications because

for that use we sort of to spend boxing take mechanism and how that would

work

is pretty interesting

a long time ago i helps port S U X to daddy and so i

have a fair amount of expertise in that and i think it makes sense to

use the ceilings for part of it but the portals work is sort of essential

for user interaction pieces so you have a sort of unchanging applications must never do

this type layer and then sort of maybe more enjoyed or i was like dynamic

application asks to do this type of model and that's portals

but i want to do this we really need system the in the session because

it enables grouping binaries for an application like right now one things this sort of

core part of the known three model was it's application focus and the way we

match

the X window created by nap to the process is actually just really horrible and

like to move away from it so getting applications inside to see group we just

P a fundamental enabler for moving towards this

O S application split

i love to unify the idea operating system extensions of stuff like proprietary and video

drivers what i think i was and always extension the package model socks for this

because obviously what you really want is you want to have your system and then

you say okay i'm gonna stalin be a driver and ineffectively rebates whereas right now

the package manager will say okay well this package needs this and what you upgrade

when with really important is to enable someone say upgrade the base toss up in

video driver try nouveau does the new version working not have the package manager you

know complaining over here so that's a know it's extension fonts that sort of stuff

are you can plug ins or was extensions and unified those with than so extensions

so in conclusion

core qualities important free software is very important to me but

quality is basically job one if we don't do that

it's not really doing anything

my provide same application framework and i there is the science just the

the core how her they distributed just making it easy to

getting exposed to the auto tools really rough for a lot of application developers and

so i was still allow people a package thing so concretely applications just still have

a make install type stage i think but it's

the build systems really do need to of all to the point where

it is possible and easy to read an application without being exposed to all those

words

and of course absolutely essential to this is just maintaining our links with the various

downstream projects like that in reference links i mean all that stuff that was in

the beginning but i have again tried to focus this is

it's not i'm not trying to be daddy and so it's just focusing on a

quality so making sure that's good

and then feeding changes elsewhere

and then if you wanna know more about the project these are some knowing less

and i'm usually available on our see any colours should be a male

so that's it

i know i kind of blew through this but there any questions

okay

sure so the question was about don't reproduce ability so particularly like are the binaries

identical that so

okay so all the binary stolen use repository i didn't really answer this question is

probably to talk about what this other project nix O S does this one ones

i was looking at one as do research for this project so next does checksums

to but they do they check some all the input to a bill so for

example if an environment variable changes that outputs a different checksum obviously and they're obsessed

at the stuff but there's a lot of things that sabotage that like python P

Y C files have the timestamp of the file the referencing which is useful useless

my system it hasn't been in important because i've money a disk space

so

so okay sorry i see we're more as you know story each generated tree is

the manifest a actually what i call results manifest so if you remember back is

this

back here so the result manifest is like the hard but be part of buzz

example word has the exactly provisions of the get repository that we used to build

the street that doesn't necessarily make it reproduceable exactly because i'm not putting in the

manifest what the host whole system uses open embedded is a cross compiler crossed environment

so i start format having for enterprise linux for which is what spinner that number

are runs and then across builds G C from that so if you want to

fully reproducible build i'd probably have to put in the manifest built from whatever and

that's something i can improve

and back on

okay

i still i could be loud enough

so you say you're not rebuilding like when you rebuild you decay you don't rebuild

web

right exact you have like a stuff throw weight might be considered of probably method

of doing that of saying you know i'm not going to keep that will result

from this but i would like now that i know that i changed you live

and we run integration tests later but that only it says

whether the binaries you produce this point i'm work against

the current that you would you have installed not whether they will have and we

build a that's right and i have had problems with that like a completely right

now so name under a little annoying like when cargo bonsai name i have to

force a rebuild of things and so the way that works is at the bottom

i just have this thing i call bill that thought so if i updated version

and i say nine type and say gee the G T K three here that

will just force to rebuild of those things a build server will just to me

it's catches

and i do that well as you can see twenty times in the last six

months which that's pretty reasonable to be perfectly honest it's worked well

okay

how do you put and it's also am doing maybe analysis of the point is

to make sure that we're not we can maybe a

for sure

it's kind of helps a lot the integration test phase helps a lot so the

in the i am to you will broke well right now we would need though

we would run binaries that you live itself bill but if

for example i would know if you live broke in the and weighted probe G

T K

as long as they didn't both get rebuild and but as far as more stack

analysis type stuff like i do your thinking me like upstream tracker dot org or

whatever it like they have some or and it this beta the less expose and

you pick a tent it's a be a

being able to not this

okay

i mean even if a test or something but we criminalise between editions actually idea

of a big difference their previous symbol question two signatures like right well course

the libraries themselves do have maybe i mean do you live has a list of

export it simple so it's very difficult to break you pi thereby

i mean it's a question of what we do with the data like i could

generate that it but what do we do with that you know do i try

to go back to you know daniel and say okay you about you probably decimal

to maybe i don't probably be very useful for bias

i don't know i just haven't

i it hasn't come up i guess they

so that when you know just them so just when this question and then you

can go and that's your questions if you to quit here

i can build a what's my question which is weeks scroll i will be break

something

so see you at the moment opinion of ceasing you bring can exploit so on

a scale

right

i don't know i mean it can obviously big claw everything jasper watch is the

output to and you know he looks at would make sense to say it sent

me email

i could well

i think that was potentially come out of automated bisection because the thing is things

can break for reasons that are not careful that happens pretty often like system he

has a very complex build system and tool chain bugs sometimes just bite and

you know is it a tool chain issue "'cause" they use some F function sections

but it doesn't matter but basically it's not always there for the be like you

need a human site actually a stipulation i mean

but i certainly could harvest email addresses from the get repositories and

i mean broken that it doesn't usually break that off one email that i maybe

it's well it's funny like some days things just break left and right and other

times it'll go three days and i just don't even touch it and just finally

right i think that's it right thank you all for coming