alright so
i'll just get started and what people you felling in if you come so i'm
gonna be talking about this project called know mostly it's sort of an awkward name
but donna names not too important
before i get started here though before talk about what i think it's really important
to talk about why am here why i care about what i do and the
most important thing that really drives me is free software very passionate about it and
i just think it's
it's always been important sense ever since
which is a men road
started the G P L and before that secondarily i just have a lot of
fun working in this community and
you know hang out with other people in interacting with before the world it's also
but fourthly right having a prize linux is very important to me and know was
a very important component of that had enterprise linux
and so when i ship
this product to our enterprise customers i wanna know that known is providing red hat
enterprise linux with high quality software
so all these things are important all these things drive me
so secondarily though it's important is to like restate or assumptions briefly why assume here
well this is all from the website obviously the free software the software freedom part
is in common with michael's the other ones are important
but i wanna dive into this connected aspect "'cause" i think that's really interesting part
it's probably the most interesting to me so this is sort of the way i
the way i think of it is gnome is in ecosystem and we have our
centres of gravity i'll talk about those and then obviously we interact with all different
kinds of other ecosystems so greater or lesser degree looking at this list
you know obviously we share a lot with the canoe project tries in is there
an use this on components right enterprise linux very important to me obviously uses a
lot of know and there is there's
the programming languages icons for roles monos
and other ones like open in embedded which in turn are linked other ecosystems you
know open about is going to wind river and all these other embedded devices and
stuff like that
so
it's important is sort of just visualise
there's all these ecosystems around the gnome is there around the centre grab these that's
the way i think of it
so
with that background what a minute talk about is just to build system
but i'm really excited about it which you guys made notice if you see it
online
because i just kind of always wanted to send it's the in so successful now
so
my goal with this bill system is to focus on owns core or centres of
gravity and
not exclude other things but just make sure that we're doing well what is important
to us and all talk about that and the continuous aspect so it is building
but it's also testing continuously and i'm i could talk for the rest of this
whole session about continuous integration why it's so important but i'm just gonna gloss over
that
so
it's testing
a lot of get repositories a lot of the what makes up cornell so kind
of when i started this project i start from this is sort of minimal system
and build up
from scratch and so i got to know him shell like i wanted to just
minimise interactions with other things so we can again focus on our core aspects so
striking a lot of get repositories for example all hundred and some X or get
repositories most of which are basically just the protocols and never change but
and of course all the know network the think that are important network manager system
do i wanna make sure that known works well with the latest system so tracking
system do you get master very important
the basis done this is sort of stuff like that i just don't care about
like for example an S P R and S open ssl those all come from
of an embedded and basically are ran make some binaries popped out if you're to
work that's about the level of Q A that i gave to the
also crucial is that i have
a sort of course focus with this minutes only targeting virtual eyes environments because
there's many reasons for this one is that doing testing fertilisation is whole much easier
than trying to have like a test lab wired to physical hardware that's a whole
other level of management personalisation it's really easy i can script using you make it
do a dance when i wanted to it's great and of course
there's other aspects of this like there's no security updates because you basically need people
there employed and obviously my employer does this
i'm not trying to compete with them obviously so these are that's sort of high
level summary of what this project is and kind of hopefully just is important what
it is not
so the core values of known as i see them obviously we have the G
T K stacked you live is very important to me because i think it
it's a fundamental pieces for writing modern applications it has the event do it has
a file system abstraction a bunch of stuff that really is a core part of
an operating system and i'd like to see it used more particular in red hat
enterprise linux
also in known we have accessibility internationalisation also part of our core values and he's
the secondary list is kind of my list
software engineering very important to me get the all the basics of doing continuous integration
you know just are we leaking memory you know just
again the basics and i'm trying to sort of push for
split between the operating system applications and then always exceptions and i'll talk about this
a bit more later but just to repeat again the goal the system is to
focus on gnomes core values and feedback code continued a feedback as we have been
doing for years back into these other you can systems
so
the non goals no enterprise features no packaging know enterprise features there's no you know
if you want her breast for example that's not something that you know i'm gonna
pass
and a fast home again not something about it
testing G T K G but
so what has a system achieved well it's been online deployed in the known data
centre since december ish and since then it's been building fairly reliably and very importantly
without human intervention i'll talk a little bit about the details of how that works
but a lot of times it's just building one off
doing something else and it's just working fine
but more importantly than the building it's been doing testing and all of talk a
little bit about the details of that
i have some numbers later so this is something that i'm most proud of is
so but i'm from a get pushed or something like do you live in so
when i'm booting a virtual machine is down to minutes now and
it depending on the type of change
like
depending on the type of change i think i could get that down the second
so that's always been michael
it's totally possible the other thing that's really important to me and i talked about
this last year a bit but
the story for how upgrades works is just crucial for no and
something i just find deeply embarrassing about the current red hat enterprise linux six which
i run is that the upgrade system just doesn't know anything about the browser in
just it's not safe to upgrade it like you know if you're if you lose
power halfway so part of this project has actually been redesigning how it this work
and so using it was three system upgrades actually fully atomic you can pull the
power well they're running and it's just sort of crucial
so even more interesting there's a lot of tests that right now so before developers
kind of ran make check
periodically manually now have automated test automated make checks of a lot of different models
there's
three hundred fifty test binaries which in turn because of G test are in turn
out a lot depends how you measure but this takes a couple minutes all talk
about that later another cool thing is that for a long time unknown would really
test G D M that was kind of ray would run makes you will make
install in his laptop but it's not something that fit the J job model right
J scroll only affects your session and so sometimes G D M well it was
kind of broken but now if G D M breaks all know in minutes ands
that's been working like
so i want talk a little bit about the always three system and how its
kind of different from the packaging model because it's
pretty it's pretty crucial to how the system works
so i buster itself is just kind of like a user space filesystem enter and
i like the term get for operating system binaries
it's misleading in some ways but on the ways very accurate so what industry does
is it manages to let you have let's you more parallel blue multiple operating so
multiple versions of multiple operating systems
without affecting the physical root directory of your operating system
so
it's like get in that it's a content address system so it naturally to duplicate
so if you have multi same multiple versions of gnome installed they only use space
a by proportion how many files are different
so
i'll talk a little bit more about that later to it's different though from a
lot of image based appointed systems that happen prototype out there like red hat enterprise
linux have this state was project for a while which was basically a kind of
image of a system where you download new data to block level and you're at
sea is N F S mounted or something like that all those trees different from
that because it does operate at the same level as package mentors do but unlike
package managers it just a done content replication system in this allows a lot of
neat tricks i'm a show those the next slide but straight the way to think
of is really am your image of packages are lots of little filesystems with some
shell scripts that run as root attached in their upload in the internet and you
assemble them on your client machine string model you replicate filesystems from a build server
or you can generate them locally
and it is filesystem independent so it does leverage some butterflies features mostly through G
live you we automatically use the better fess i aukro for a copy and right
type stuff but i think it's sort of important for general purpose system to work
on top of plane yet see for and that's actually what i test on the
most because fifty fours really fast
so
it does have a format for installing on your local system but it also has
an archive format they can run on a web server and that's sort of how
this the build system works as there is than almost resides the build system it
commits to noticed repository and then you can download it directly from just a plain
old apache
so here's some tricks that i wanna make possible and i've been protecting this a
little bit with laura is so i run redheaded present X six on this laptop
because people favour subscriptions any bill fix it but it's obviously important from able need
to be able to run known get master because i wanna be able to contribute
to it and make sure that it still works for the next version so
one the way i plan to have this work is all just have reading representing
six am i root partition and all street just sort of installs inside that without
disrupting it so i can have the know most re what's called build master just
the thing is tracking to get repositories but you could also parallel install multiple versions
of or a very easily
the hard part about this is fixing or P M to not be the lowest
level anymore so that's still in progress but i do have the point where you
can have the base system and then to the newer one and test it working
fine
and when i say five hundred builds here that's really possible because it just comes
down to how many files are different and between
things just don't change that much and there aren't a lot of big binaries so
you really can have five hundred builds installed in more interest a need or trick
that you could do something like automated bisection which is where you know you have
your starting point you say okay i get graphics corruption recently and sort of write
a test binary that maybe just takes a screenshot what for the corruption and then
just keep booting into all these bills and so you find the build it's broken
and from there you can from the source code i really need poster trick in
this is kind of part of how the cork you a system works as it
can up update virtual machines while they're entirely offline
and this works basically because it's a don filesystem replication and not like packages where
you have these ritual scripts attached and what they do
was she has none of that and so that's this is actually a very useful
trick in general and like i mentioned before i'd like to see austria eventually slide
underneath these package systems like D package in our P M and all that stuff
where or she would give the parallel booting the atomic upgrades that sort of stuff
to them
so know much three is layered on top of austria and it's very simple there
when i say there are no packages i mean i went through the entire system
in eliminated the concept at every level as much as i could to try and
gain reliability and sort of optimised for
automation
so than almost relay or has this thing called a manifest which is literally just
a list of get repositories to build and work
it's very simple and as a consequence of that simplicity it has turned out to
be that has six months of and very reliable promote stories and so if we
look at like example of what the manifest looks like well that's multi one that's
the entirety of what would be a package in my model it's just it's just
to get repository insert it'll to build in this order and that's it that just
means track it master of lib X M L to
and spend enough for quite a while
the component about one i'll talk about this on the next slide but basically that
means this is only for the developer image i can also choose any get revision
to build it anytime unlikely package in our can none of this cares about version
numbers at all so i can be tracking system to get master and say okay
i can no longer log in and the current version which has happened so i
say okay all just tag back to the previous revision builds done like within minutes
it's completely on like the package systems because they have this deep obsession with version
numbers that
permeates every single layer so none of my system cares about version numbers all any
get revision at anytime
and is the spice example is just how configure options and patches work i originally
was trying to only have get repositories and no patches just to force myself to
feed changes back but i sort it's too impractical so i ended up having a
couple patches
so this is something i think really illustrates just how different it was tree model
is from the package thing so on the build server now it's been doing continuous
integration since december right there is over six thousand complete filesystem trees like i haven't
lost any history right so
in the only occupies sixty gigabytes and this is sort of the power T duplication
so in theory again not only can you have five hundred bills locally you could
download in entire history of all the bills from the build server and
you know we'll get those all have links to their sources inside of them
so
i just give a lot of flexibility right you can have a sort of a
system and then add remove stuff we're supposed trees all about the files so the
way all history is sort of designed is that sort of having a bunch little
packages you have full complete systems that share space and i would you have multiple
versions of say what i call runtime tree but you have a basic runtime tree
enough to run out and then i have an image div L with all the
developer tools and these just naturally share storage so us three let's you switch between
atomic so i can i normally run the developer imagine my D Ms but you
know sometimes we've had boards where things because of obscure issues only work in the
developer image like a they in package world they needed deborah develop package installed i
can easily just switch back to the runtime tree which effectively hides all the developer
tools so
parallel installing multiple file systems and then besides all the for variance is actually three
it's a the build system outputs a pipeline so is what i call build master
which is all these get ready all these get repositories get bill integrating the build
master all that happens there is there just build a just take the latest get
run make install duster and it's in the tree you can download that that's very
useful obviously but
i don't expect many humans to want to run that directly so there's a secondary
stage which does what i call smoke test and this just updated disk image puts
it union does this appear to let you log if it does that gets attacked
just like get repositories get tagged
and there's a secondary stage which is integrated is that after all the make czech
and stuff like that works but
the way i sort of vision this is there's a like a long pipeline you
can imagine of the very top we have build master and then something like red
hat enterprise linux is like way a down there but it is a sort of
long pipeline of extended increasingly extensive testing
but it still really important to be able to run the build master because sometimes
things a broken and i wanna be able to debug it locally so you can
download develop about run gtd on the latest G D M and see what's going
on
and there's also at a task the generates disk images
after something
so on talk about the story of
after you run get push to run in to get repositories that are tracked your
what happens
well as far as i know on the first person ever demand push notification from
don't get so that happens and then we basically do a copy of the get
repository under the build server
so this bill task though important thing to understand is be in a feedback is
really important to me versus correctness their different tradeoffs you can make your so the
bill test only rebuild get repository that change because concretely i can't rebuild look at
you take a every time G T K changes it would just massively slow down
the cycle
it does
it assembles a build route which means it's a clean build in this is actually
really important because some things just
it just really increases reliability but this part actually really fast compared to the D
package people there and the mocking those sort of tools it averages anywhere from five
to twenty seconds and that's because it uses hard links stuff actually building the thing
it starts from a clean get repository
so this is also kind of important because it'll for example instantly catch a missing
file that you forgot
but for something like to live that's a couple minutes and that's the same with
change builder can whatever and then the result of that install gets stored into repository
and
this is kind of like the package that is
this us the string here than almost three components to of X eighty six sixty
four that's kind of like a again branch big it's torn repository and then we
assemble all of these it just takes a union of all the different components so
i have to live i have lived to stored has their own little filesystem trees
and i just layer them on top
and that's very simple so there's no dependency resolution there's no packaging here and that
takes a minute right now that's something i could speed up but there's different trade
offs there and after this again this is just like
for and i know change of for a very small module this is around two
minutes from get pushed to when you can actually download it reliably
there used to be the concept is yes is the short answer and i used
to try and you triggers on the client side like the idea was that you
download this tree and then afterwards we would run stuff like T update i can
cache on the client
but it's i moved away from that because
then i couldn't do the offline updates on of the M and that was really
crucial so i sort of took the hit of having each multiple versions of the
cached icon caching each tree
it's not that bad and it makes up it's a lot faster
baseball for the client this is one of the things it does slow down the
build process for example on for some things i have to rebuild in a drama
fast whereas at least for their and i don't know about that the end rebuilding
the trim F S on the client every time you crouching so in some cases
i pay a price on the build server where is in the package will you
pay and it's client
that again sorry
how many different triggers
seven or something maybe like for hang go free type
i don't know there's not that many L D config
i mean i can look really but those are pretty fast execute
so
this i'm not excited about just building again i think the passing parts really important
to me so use live guess the vast amounts the filesystems non root
so the smoke test the way it works is we have a cash disk image
representing the runtime tree you know without all the developer tools and then i make
a little copy on write disk and i had some testing stuff for example there's
this service that runs in the virtual guest export the system you journal back up
to the host which then looks for stuff so system he has this nice concept
of message ideas which are
basically Q I Ds universal unique ideas that some event happened right
so an example would be system D M it's a message with a particular message
id want something court
so if someone makes a commit to say network manager that "'cause" it just simply
say for startup
the smoke test will fail
for them are patched gnome session timit a particular message id of one of the
things get manages fails like so for example if gnome shell exits with an air
or i know that pretty much instantly on the hose and i can terminate the
V M and then if you see nair see the gnome hackers are see a
red line for the smoke test that's what this
so most often it happens when people commit syntax errors in the norm shell javascript
files
lately anyways and i detect that very quickly
and the integration tasks this is something that i think is really need to which
is before the gnome release process each operand make czech kind of periodically
randomly
and then theoretically you ran make just check before release so in the know most
remodel i have all the tests almost all the tests that you run and make
czech installed in the system so i go to virtual machine and then i have
a little testing framework that runs all of the tests after nes change so it
change translation update an am show i will rerun all that you a test i
will remind all the clutter test because honestly have plenty of C P U power
and
why not i mean it actually i wanna have
you know a more interesting test scheduling type system but
the build server has thirty two cores sixty four gigs of ram in flies of
the stuff
so what that is occupied so my time for one hours race conditions on the
tests
do you live in particular has a lot of tests with race conditions and
that's
some of those a bit hard to track but fortunately most of them are just
bought some the tests and not actual core code
but i didn't i did find one conceptual issue and you live from tracking down
these
but for the most part most the test did work already because the developers we're
running just not relied continuously
so just very different
so some things that i'd love to do with the system kinda have
twenty minutes okay
okay cool so
open sousa has this really well they have a this thing called open here which
has some really cool features and some kind of weaker features but they have a
really nice but you and i love to have something like that another feature that
i wanna add is mostly has try server so i wanna have does
does this patch break known as a service basically where
i mean there's no reason i couldn't just start building inbound patches and bugs a
lot and testing on so you know one an optional developers can upload
but
bunch of patches and i just take those do a smoke test that sort of
thing would honestly be pretty fast is just the tricky part is sharing stuff between
the different builds like the get all the get repositories are out four gigabytes and
i don't wanna read clone them for each bill but making sure like and linking
currency is just what's kind of starting a little bit from this
and i would love to get into like software engineering stuff and one thing that
i've always wanted as you know just a graph of how much memory were using
on the default login how much memory using one epiphany is and wikipedia i mean
just
also the data there's no reason i can gather this the virtual machine aspect does
the limit kind of the type of performance metrics that i can get like i
can really get approach are because there's all this P N I or activity in
the build server and it would just be too noisy you really need a nice
dedicated harbour lab
probably
that's true
i think
i mean there's so many different ways to do it i don't know if you
knew has the hoax i would need for this i mean that would be a
pretty reliable way to do it i installing agents in the guest is obviously one
way together a lot of stuff but us
then of course you have the heisenberg effect where you you're affecting what you're measuring
right like your agent is using memory your agent
so
i would love to have automated bisection on the build server so you know right
now is if the smoke test fails i basically just run get log and
you know it's usually pretty obvious
who broke it and why but a lot of times it's not actually and just
like a try server there's no reason i couldn't just allocate a builder and say
okay i have this regression range well the know most remodel it's of course over
two hundred get repositories so
handling the multi repository thing with bisection is an interesting problem but again there's no
reason i couldn't do it just haven't gotten around to it
another cool thing is a little bit more blues guys just a sort of live
collaboration setup where the developers actually running get pushed to a branch and we instantly
build it so you have a some work and am shell and it designer and
that you know develop design i want to see the developers doing there's no reason
i can just okay to server that just sort of tried to do that as
fast as possible let them almost live collaborate
so that's
that is not shows the system but i have a lot to talk about where
i see the high level future the system so
obviously
okay i guess i didn't mention this but there is no application installation mechanism in
the system yet because it's a hard problem it's
very hard problem and there's a lot of prior are in that sort of stuff
there's like most all or have re pronounce it there's a bunch is click packaging
is all package on the stuff around and they have different tradeoffs parts
the thing that the thing that
is very important in this discussion i think is to draw a very strong line
it's very strong distinction between free software applications proprietary ones because i think that demand
there's some commonality there but
i don't understand why so many people in free software have this obsession with generating
binaries in caching i mean the most important thing if you're talking about free software
is the source right and who's committing to that and why and not this package
stuff so for free software applications i'd really like to emphasise that sort of stuff
like who's writing this out you know let you let you run the latest version
help you contribute to it that sort of thing
there is the aspect where
it is important to say who built it because you do yes you obviously have
to generate binaries to run it but it's still important to say who built it
so for example if you know if we have a system for applications we're doing
continuous innovation and fifteen outputting zip files or something who knows doesn't matter it's still
important say build by know because you have to trust the builder
and so i think i think the aspect of the intermediary is still important
again i would love to do continuous integration for the applications i sort of paved
and added some core apps to the manifest so you know obviously epiphany is in
their gnome terminal the system would be completely useless without it
and those are kind of
those are available but the really tried you wanna get to is where you can
remove the web browser from the operating system
apparently that's important and just you know peer review i mean this is the same
thing i'd like to see for more for the core operating system is just emphasising
the peer aspect of free software
you know again who who's doing this and why
okay
so obviously this is gonna kind of controversial topic throughout norman it really gets to
the heart of the distinction between what an operating system is and what it distribution
is and that sort of thing which is
where do you draw the line for what lives where right what might applications have
to bundle
so stuff like the backchannel to i'm not a big fan of the back small
to i really wish that it'd been but you will nice to a library before
so i
i you know do we except that is barbora base platform and it's ugly api
i don't know stuff like hire O obviously we made the decision
this is par the platform it's part of G T K spell checking is pretty
much perfect example of something where well we have some libraries out there may not
be great
whether i could should with the base system or not i don't know
and of course even if we're just talking about court know how long do we
should you take a to do we encourage do we allow
allows only should application developers still target in some circumstances and of course we're talking
to do you for enterprise and how long would we should G T K three
well so the way i define operating system is something your application will run on
for ever
and this
if we remove stuff then we're not really an operating system anymore kind of moving
back towards a distribution model so i don't have answers to these questions
a sort of punch on it and said okay well i'm just going to
to you know to make an operating system we need to have basic quality assurance
and so that's what i've been working on for a while like just fixing race
conditions and the G let us and make sure making sure that worshipping platform the
can be used to build applications and how all that works
so a lot of questions and so there this
so i think proprietary out the needs a proprietary applications are totally different
and
i mean
for one thing they
their time to an architecture usually if we're talking about me to binaries and all
of a sudden you're kind of in the windows world where you have
applications that were targeting X eighty six and then if we wanna move to new
hardware platform we can do it
i think that sort of thing are few icgs can just sort of deal with
it so i'm not too focused again on the proprietary applications basically what i want
i think it's important to enable people to run proprietary software
if but not encourage necessarily and so
i'd i'm gonna focus my time and just the core free software gnome applications because
that's again why i'm here from again that's all
so this is something that i was actually hacking on the plane
and which is basically just system D for the user session this is kind of
just a for corporate system feature but
using this we can actually this would be something that would help enable running proprietary
applications because
for that use we sort of to spend boxing take mechanism and how that would
work
is pretty interesting
a long time ago i helps port S U X to daddy and so i
have a fair amount of expertise in that and i think it makes sense to
use the ceilings for part of it but the portals work is sort of essential
for user interaction pieces so you have a sort of unchanging applications must never do
this type layer and then sort of maybe more enjoyed or i was like dynamic
application asks to do this type of model and that's portals
but i want to do this we really need system the in the session because
it enables grouping binaries for an application like right now one things this sort of
core part of the known three model was it's application focus and the way we
match
the X window created by nap to the process is actually just really horrible and
like to move away from it so getting applications inside to see group we just
P a fundamental enabler for moving towards this
O S application split
i love to unify the idea operating system extensions of stuff like proprietary and video
drivers what i think i was and always extension the package model socks for this
because obviously what you really want is you want to have your system and then
you say okay i'm gonna stalin be a driver and ineffectively rebates whereas right now
the package manager will say okay well this package needs this and what you upgrade
when with really important is to enable someone say upgrade the base toss up in
video driver try nouveau does the new version working not have the package manager you
know complaining over here so that's a know it's extension fonts that sort of stuff
are you can plug ins or was extensions and unified those with than so extensions
so in conclusion
core qualities important free software is very important to me but
quality is basically job one if we don't do that
it's not really doing anything
my provide same application framework and i there is the science just the
the core how her they distributed just making it easy to
getting exposed to the auto tools really rough for a lot of application developers and
so i was still allow people a package thing so concretely applications just still have
a make install type stage i think but it's
the build systems really do need to of all to the point where
it is possible and easy to read an application without being exposed to all those
words
and of course absolutely essential to this is just maintaining our links with the various
downstream projects like that in reference links i mean all that stuff that was in
the beginning but i have again tried to focus this is
it's not i'm not trying to be daddy and so it's just focusing on a
quality so making sure that's good
and then feeding changes elsewhere
and then if you wanna know more about the project these are some knowing less
and i'm usually available on our see any colours should be a male
so that's it
i know i kind of blew through this but there any questions
okay
sure so the question was about don't reproduce ability so particularly like are the binaries
identical that so
okay so all the binary stolen use repository i didn't really answer this question is
probably to talk about what this other project nix O S does this one ones
i was looking at one as do research for this project so next does checksums
to but they do they check some all the input to a bill so for
example if an environment variable changes that outputs a different checksum obviously and they're obsessed
at the stuff but there's a lot of things that sabotage that like python P
Y C files have the timestamp of the file the referencing which is useful useless
my system it hasn't been in important because i've money a disk space
so
so okay sorry i see we're more as you know story each generated tree is
the manifest a actually what i call results manifest so if you remember back is
this
back here so the result manifest is like the hard but be part of buzz
example word has the exactly provisions of the get repository that we used to build
the street that doesn't necessarily make it reproduceable exactly because i'm not putting in the
manifest what the host whole system uses open embedded is a cross compiler crossed environment
so i start format having for enterprise linux for which is what spinner that number
are runs and then across builds G C from that so if you want to
fully reproducible build i'd probably have to put in the manifest built from whatever and
that's something i can improve
and back on
okay
i still i could be loud enough
so you say you're not rebuilding like when you rebuild you decay you don't rebuild
web
right exact you have like a stuff throw weight might be considered of probably method
of doing that of saying you know i'm not going to keep that will result
from this but i would like now that i know that i changed you live
and we run integration tests later but that only it says
whether the binaries you produce this point i'm work against
the current that you would you have installed not whether they will have and we
build a that's right and i have had problems with that like a completely right
now so name under a little annoying like when cargo bonsai name i have to
force a rebuild of things and so the way that works is at the bottom
i just have this thing i call bill that thought so if i updated version
and i say nine type and say gee the G T K three here that
will just force to rebuild of those things a build server will just to me
it's catches
and i do that well as you can see twenty times in the last six
months which that's pretty reasonable to be perfectly honest it's worked well
okay
how do you put and it's also am doing maybe analysis of the point is
to make sure that we're not we can maybe a
for sure
it's kind of helps a lot the integration test phase helps a lot so the
in the i am to you will broke well right now we would need though
we would run binaries that you live itself bill but if
for example i would know if you live broke in the and weighted probe G
T K
as long as they didn't both get rebuild and but as far as more stack
analysis type stuff like i do your thinking me like upstream tracker dot org or
whatever it like they have some or and it this beta the less expose and
you pick a tent it's a be a
being able to not this
okay
i mean even if a test or something but we criminalise between editions actually idea
of a big difference their previous symbol question two signatures like right well course
the libraries themselves do have maybe i mean do you live has a list of
export it simple so it's very difficult to break you pi thereby
i mean it's a question of what we do with the data like i could
generate that it but what do we do with that you know do i try
to go back to you know daniel and say okay you about you probably decimal
to maybe i don't probably be very useful for bias
i don't know i just haven't
i it hasn't come up i guess they
so that when you know just them so just when this question and then you
can go and that's your questions if you to quit here
i can build a what's my question which is weeks scroll i will be break
something
so see you at the moment opinion of ceasing you bring can exploit so on
a scale
right
i don't know i mean it can obviously big claw everything jasper watch is the
output to and you know he looks at would make sense to say it sent
me email
i could well
i think that was potentially come out of automated bisection because the thing is things
can break for reasons that are not careful that happens pretty often like system he
has a very complex build system and tool chain bugs sometimes just bite and
you know is it a tool chain issue "'cause" they use some F function sections
but it doesn't matter but basically it's not always there for the be like you
need a human site actually a stipulation i mean
but i certainly could harvest email addresses from the get repositories and
i mean broken that it doesn't usually break that off one email that i maybe
it's well it's funny like some days things just break left and right and other
times it'll go three days and i just don't even touch it and just finally
right i think that's it right thank you all for coming