say hi everybody thanks a lot for coming on to my talk about global
so i'm robert right guy work for intel open source technology centre on the kind
maintainer of the whole project
so for those are not familiar with probably it's other consoles gpu graphics library originally
created for the cluster projects they are these days as a standalone project
it was created to deal with the possibility across different versions of opengl so course
opengl yes opengl and even experimentally a bit these days with the web you know
it's is so dealing with portability across operating systems
one of the differences with some other graphics if you refuse are base is that
initially when we created it we were interested in supporting more multiple sort of orthogonal
components being of the contribute to the graphics rendering in a way that didn't trampling
each other's us take tracking
so in a still come hoping to go through first a run through of how
we submit and process geometry with a gpu and when the something out
and i want to then give
an overview of the utilities in the couple library that try to make it really
practical for writing graphics applications
be on just access to the G P your hardware and then talk about some
of the specific things we've been developing more recently just say that people get some
insight to how the project is actually developing
so first so wanna talk about
basically the lowest level api even in cover which most closely matches have a gpu
process is data
i'm kind of personally interested in trying to expose more people to understand have a
gpu itself works especially in the context of user interface development where i think we
try to be on the utilise the flexibility of this hardware
and the state-of-the-art for you are is today this is quite basic in comparison to
you can see to sort of game developers and how successful they are using this
hardware with many of this and that kind of constraints is us with
interactivity and real time graphics
so
anyway the first thing that any applications gonna have to do they're gonna use google
to draw something are with the G here is great and of the cobble context
this is gonna just you could cool context news is gonna create your sandbox for
all the data the call was gonna need to track for your application
so then we're gonna need to create a so it's a destination for what we
draw remove are rendering this is called a frame buffer
a frame buffer is a collection of to debug is
the but we care the most about is the colour buffer and that's where we're
gonna store the results visually of what we're drawing but a frame of mind also
contain other ancillary about this so that are is
for accumulating the depth of the geometry that is used during this that we have
to see so you know these guys
in probably create a frame buffer you can you can great and on screen frame
buffer that would be visual to a visible to use of coding got one screen
you with the size alternative you would create an offscreen frame buffer by creating a
text you have with a given size and then you would rat that texture with
as an offscreen frame buffer
so if we have a destination that's where we're gonna store results then we need
some data that we're gonna process with the G P A and how does a
gpu except data
so
and gpus
also actually the is so the thirty forms of data so we i think
so
the in cover we support lighting image data have for
sorry
in the order
and image it's a as a form of a so that we would expect we
have person a gpu be on search german tree and it's a in probably support
lighting textures on the from finals from afar name or from about four with a
given size that and by sending data on that texture us one is being up
to images into a larger at this of images a
and it's often useful to be able to
how can we just again as a high level protection the way you often it
is a compact and to be have the packs more images into a larger images
to avoid state changes and another form of textures the gpus process of pictures great
from when your system resources such as X eleven expects or while and
i'm sorry the older this
okay
it's just that
the left and right click it doesn't very this is a real andreas breaks the
slide down to the sections which of vertical sections like this pressing that right skip
saver
pretty big chunk that so which
so about a so this'll spree this slightly better slide talk a week or two
meeting date to the G P so depressing we need we have to do is
the actual geometry into the G P A so first we need to actually understand
what they to the G P processes and that is geometry which is test selected
from triangles is the primary form in which we price estate on the G P
A so you
to understand
you have the how we describe astrology the next thing to understand that's a this
up about that is the stuff like just jump ahead at and sorry about that
but we're gonna fast go through explaining how the geometries this but apologies and
so the way we describe it geometries might tell is german treaters trying plated from
attempt to assign different triangles and there's triangles themselves made of that but to cease
and that is that sees are themselves components of what are called attributes
so examples of attributes he would be so positions for each vertex in a triangle
so you would have a position
which could be a coordinate two D or three D coordinate reach but text in
that geometry or it might be a colour for each vertex in the geometry or
it might be i
a texture coordinate that would could be at to detect record there
it is the mapping a specific subregion of an image that is a don't you
there's a lot of flexibility in the gpus to what data you can associate with
about season your geometry and these are just very the examples but it's you can
have all of the cost and data so i see the outcome actually
but they don't you can also have many different apologies so tip about apologies would
be that you have a triangle list which is where you would have
so i sets of three but the sees which represent completely disjoint and disconnected triangles
or you might have a strip of triangles and that would be you divide your
first set of three but sees feel first triangle and then one additional verdict is
gonna connect to the last two that sees of your script to make one more
triangle and so one more vertex connects the last event these answers a more data
efficient way of describing some geometry than with the less if that's applicable to your
which on your trying to describe and last example would be a band where you
just have a single but clicks in the centre and then the rest of the
triangles gonna spend a lot of that
and that's and one
so how do we actually described estate using the called I P I
so the first thing we're gonna do is describe
decide what kind of data we want to associate with the each of those matrices
there is some different strategies as to how you present estate to the G P
so i'm just gonna give you one very typical like how we would do this
so
we decide that we wanna have a position the colour and texture gordon associated with
each vertex in our geometry we're gonna create a since even greater about six am
i-vector extractor in this case and it has a position it has two components of
the position X and Y coordinates into D and there of type fly we're gonna
have full components of the colour but we implement out for a and each of
those is a by a number gonna have two components protect recording it and those
are touch like nist one
and
this but what about an this is just that we've taken on my vertex struck
which is a vertex i'm with declaring an array of these and we're gonna plot
out we've got three about sees for one triangle and with plotting out the positions
of the but sees giving each method the colour and where specifying the texture coordinates
you
so we nine to up like that data to the gpus able to read and
prices that and that's just a case of uploading into a cover that should be
but we do that by coding a battery but when you repulsed in the size
of the data that we want to apply time we pass in the actual point
to the data
and that's gonna give us a reference to about
the next thing is that we're going to have to describe the G P or
how is it that it can what we got data needs to know that i
have to that data so it's right three the but sees to process it
and so if we reach out to be we need to create a cover that
should be and we go global attribute new and we're gonna pass in a reference
to the buffer which is a date there are actually so we do we declared
and we're gonna give each attribute and nine we're gonna specify what is this be
and the number of bytes that we need to step to be have to move
from the best the text of the second but sex and so on
and then we're gonna need to specify what's the offset from the beginning of the
buffer whether gpus gonna start reading for that should be here to
process that data
lastly we need to specify the number of compared to data have you have to
take
like two components
so assuming we do the same thing for the colour and texture coordinates we have
for batch we recovered attributes now
we can now define one or more primitives
the referring to that should be right so
so at this point this is where we specify the actual topology that we want
to that we wanna gpu to use to interpret that data so lists triangle friends
trips or whatever
so with the funny primitive here way because it's just a single triangle that apologies
pretty meaningless so we specify that there are three but sees the belong to this
specific primitive
and then we specifying the actual attributes the belong to this primitive and in this
case we using this of our function takes a know to make the list of
that should be says
so the
other thing that gpus comprises besides the john geometric data that i just described that
is that they can also still image data as textures citizen where we were a
bit outflow of their own and i started going through
how in cover we can like text data so we have we can let them
from files we can load a texture just as an empty but and fill out
from the C P with the point to we can also create these textures which
are cool data structures like an example of a high level texture the pose able
there are numbered if i don't use
but it's sometimes useful for this band of packing images one texture
to avoid
like a changes for
you are use cases
and might these pictures from when the system resources
what is
and
so we probably talk we put off right but what we're gonna store results with
what our data which is these triangles and images as textures so now we need
to figure out how to describe the pipeline by which this is all gonna be
processed
so i'm gonna give a really sky high level view of any it example of
how you can configure gpu pipeline to process data of modern T is pretty don
flexible and they're a number of stages a cut optional state is really but that
i'm not representativeness example but it is still a very typical a common example is
for is this technology
they're gonna start with our geometry we coming in on the left
and we're gonna bust going to this but expressing stage in this stage we're gonna
transform a german three we're gonna scale it rotated translated we're gonna projected so that
points at a button away from the origin are gonna attenuate towards vanishing point to
the
realistic
and at the end of the stage all about german tree is gonna be transformed
into the coordinate space about frame but
and it's conceptually a range within a scene of other geometry
bands
the next stage after that is the frogman processing stage in the G P is
gonna go through each of the triangles in our geometry and it's gonna i would
like that with the pixel grid about frame buffer in it's gonna consider for each
pixel where the centre of the
what's called a fragment where that is within the bounds of that triangle then it's
gonna run a program to basically calculate colour for that fragment at this point when
you're calculating not colour
you can but that's of those textures the we floated
and
also the coolness of each triangle where we have those attribute values coming in there
is can be interpolated say that for every intermediate fragment network processing we have interpolated
attribute values to contradict
program
so in the final step
we take all of those fragments and we know need to humanise there is onto
the existing contents of the frame buffer
to build up a whole scene and it this point we might also be reading
back insisting colours in a frame buffer to be able to say if the effect
of transparency by combining in coming
destination ones
and so the buttons wasting the from present processing state is there is programmable stage
is very flexible the planning stages tends to what is a much more limited functionality
done and fix function of us still a can lay down do that stuff
and
and the well in which the actually the hardware prices this is very paralysed there
is if you have a primitive without one for triangles coming and there's not really
any bordering guarantees as to have a gpus gonna price essays triangles
and it's gonna be doing lots of geometry the same time and similar with a
fragment processing the order exact order in which is gonna priceless fact is not something
you get to guarantee and it's very paralysed that you will be doing
sorry
so kind of have a gpu prices to state a high level and
i don't have the time to explain lots of examples of using the cover if
you have to describe a
variations on configuring E D U but just to introduce you to the couple plot
line object which represents that whole processing part by the G P A
and we can create a peaceful pipeline by conical pipeline you
and to just give one example thing of for figuring it where we are introducing
a texture into that pricing but we're gonna associated texture with what's known as an
idea and this idea of layers use during the fragment pricing stage where they can
be combined and
us associating attention with layers there of that
so basically now we have all of the bits that it takes to actually draw
something a process similar gpu i'll just mention frame buffer of data a primitive we
got a processing pipeline we can call primitive draw busting the reference to that data
that frame buffer and is that the choice of five line
and so that's basically that's basically it but introducing datum pricing G P A and
it's
it is to distill as much as i got time to be able to
i give you
so the next thing i wanna talk about the utilities at all the whites that
go beyond just that lower level access to the gpu hardware that trying to make
it actually really practical to get stuff done is just basic things a little mice
all graphical applications need and gonna start off by introducing the math utilities that come
with bubble
we have matrices api matrices of the way that we deal with describing transformations of
geometry in combining those transformations
just essential stuff really matrix not build on top of that capability and kate to
the number of applications that want to organise that the geometry into seeing profs
so it's very common that you want to basically start these transformations together is interested
in seeing trough pushing them onto the stack "'em" popping them as you move around
the scene rough and this can say you re accumulating lots of transformations about this
is the on changing i frequency
and that is it just
you know use of all kinds of things the next can be to have a
you know it's on and dealing with that is
rotations there are several rotation types that we have been a couple user an example
of a compact representation rotation comprised of three orthogonal rotations
and quaternions or another format it's an encoding of an axis workstation and they have
the nice property being able to interpolate really smoothly between the have the pros and
cons and but especially with quaternions
they're pretty some pretty fancy matt's behind the it's not
every time you need
and colour utilities so being able to convert from rgb take you saturation like two
spaces
other
which i just snippets this is the way that we experienced the programmable parts of
the program will parts of the gpu
this
this design say that you can have multiple components contribute into the i roll processing
of geometry and it so cooperating within one framework so
the way this works is that we have this idea that points in apart line
so examples would be vertex processing stage where you wanna be able to
make some it's you want to build on the people processing of probable it but
introduce some additional transformations for example you can provide a small snippet of code which
is using the G L of cell shading language to insert into one of these
have points in a can either replace that whole
section replace any people processing
or complete and or pen so that it can cooperate without science
and similarly for fractal processing like and set to
and that supplied to
and looking up textures and there are
places as well
clipping lots of applications once we have to restrict
the range where they want to restrict where the influence of drawing within the frame
buffer
to a certain to D regions and
so if in cover you can click to ask align rectangles transform rectangles this interactive
a battery primitives to D pass describe with fancy is awesome lines
and similar to the matrix that we consider the you have seen rockies cases where
you want to be able to start these on top of each other is interesting
trough and effectively region is the intersection of all of that is the company push
regions
rectangle patching especially when it comes to user interface is a currently very dependent on
using tech should rectangles as the
so building block of most interfaces and because gpus are not really very effective a
practising very tiny primitives one of time just to send a couple of triangles for
what within separate draw close is
is not and make good utilisation of the of the G your hardware
so it
with this there is the if you well within certain limitations i for instance with
if you little your textures into atlas textures
so they can all be referenced as part of one two like one month one
primitive
and if you limit yourself to certain types of clipping then in a lot of
cases actually couple can batch almost like whole scenes of rectangles into a single goal
cools
it's quite useful
so don't have a bunch of optional sup libraries that come with problem as well
i mean for instance we have come up anger which enables us to rend text
laid out with that anger make you know i
we have hope a which provide a test like to have a teapot so describing
shakes with bessie is lines and curves and being able to get a retains object
representing that estimation of that and being able to redraw that without incurring repeated incurring
the cost a test selecting that geometry
how would you nist two thousand is to
easily integrate a just
regular gas to carried into the a couple application and managing the state
between nice
to it you know it's
i would use T is a recent addition which i i'm speedier integration
so couple based on the G stream a framework
so now ago three some of the much more recent things that we've been developing
just say you know idea of where the focus is a and where it's going
so that's the yes i thanks to find a at the beginning of this year
we introduced a couple frame info right you know i
and it's this is about being up to get asynchronously extra information about frame is
completed so for instance when we went to remove
also possible frame of the impostor to be displayed
and it takes some time before that can post a response and gets that frame
actually displayed on to the monitor and we'd like to know too well what's the
time stamp when that really hit the display and get that but we have location
so that it can be a bit smarter about our rest is animations
based on the prediction or when that frame is really gonna hit the physical display
see you have these
i think
thanks to kind of stay in for contributing offence compensate pi two call this allows
us to insert marcus into the come on stream that was sending to the G
P A
so that we can have a bunch of what that we're interested in knowing when
the G is finish processing this way
since when that work is finished then we can get a call back into the
application to be able to
i respond to that and the use case they had this was for video integration
where they wanted to be have to make sure that
but the gpu had finish any something the need to do a certain textures before
was handed back to the video decoder library to avoid conflict
thanks to i mean a man a lot of one of the interns so we've
been working with recently and saxony a lot that's the supporting i'll at this work
for G S T
on speedier integration to couple and it's it works
particular well with to pay particular attention to making sure that it works nicely with
the state i the eyes and with multi texturing side it is a lot of
freedom to
i will all kinds of your own custom affects and things that will be involved
in a rendering of a video frame so if you wanted to do from processing
like be saturating contrast brightness that kind of thing and trade is you don't wanna
have intimated rents we listing so you need a cooperative system to allow you to
combine these effects
one
and as you have the use cases this was for project called break sure
mention that
so what you have is
another thing that were exploring at the moment there's a project that where this using
couple that i'm working on who rate which is i you are rendering engine and
provides interactive tooling for designers for improving the web design where were exploring more advanced
uses of the G P are in the context if you i development and it's
we're interested in supporting a number of homes and even we're interested in exploring supporting
web applications
it's an expiration price my remember where i will be possible and okay
so what's next
so something that i always stripe in the way that
in terms of maintaining the project and driving a project for this is ensuring that
the feature what we develop
it's really driven by the projects work the requirements of projects are actually using so
we don't wanna be planning architecture astronauts just coming up with fun stuff like that
we wanna actually have a reason why we develop any particular feature so the ounce
really this question is to consider the people using the cover like my
the for new uses a couple will take it and you directions
so some of the current users a couple that implements it
we have a lot to obviously are the do the first user couple
and it's even today the that's a influencing certain things for instance the high dpi
stuff so i guess they will need to make some small changes to the called
tango at are to consider a
the size of clips that we tool
but there is a special use case of a couple especially when we consider the
recent web two wraps
name show that way to support and i'm show
and that the bonds a bunch of the feature well as well in kabul
well who is not something i'm working on activity at the moment but it so
i wanted to mention it just because i did put quite a bit of work
into this a few years ago and it was an interesting project it really do
that should drive quite a lot of
thank you think over the time
and this was adding a gpu acceleration back into card was taking a different approach
to carry G L and are generally have said making some different tradeoffs
but it did some interesting things to do if retaining the translations of parts and
trying to be a little bit okay well about using the fix function possibly gpu
for gradients and
the be a bit more efficient
i would say much about you best "'cause" i guess not many people here about
that but you know robert's writer back and for E best the
i
this is like a couple years ago we played around this is yet not really
got serious use case but it was interesting the time because and there's actually some
slightly nifty things that you best dates to do with matching actually which we were
interested
"'cause"
so
and so a much more recent projects that working on this week which i mentioned
just briefly it's a okay and i don't hide for this project to be quite
a big driver a couple going forward actually
that mission here is to really explore in the context of user interface development how
we can use more features of the G U and how we can also optimized
the web fly between design is prototyping but at the moment without many constraints this
to the technology used in the end and
creating interactive tools
the hell optimize are what are even connecting we to sort of devices i network
and such
things
and
so
right about that
big driver
so those of the project striving that's night it sort of update for the projects
and so i'm entrance it's in it if anyone's interested in this kind of thing
graphics processing and it's especially a
why and this audience in the context of user interface development
i'd love to invite people to learn more about the G P A
and contribute to call i've either directly or through downstream projects
and creating a more that downstream projects and we
we called wants
so join us about join the mailing list get involved in discussions during this and
i have see we
created a channel for call dedicate the corpus the other day about time really just
the program
trying to send a message on again off together and say for us on better
if you just want to i project and there's a website now
find what you what you will we we're gonna probably try and i mean it
but the labour office i dare easy
which will time
nine what to make it as easy as possible
so
that i'm sorry
any questions
gonna task
the this building
what this piece to me
well are you use the
even if you're using name show the patching that it provides a would make it
more question
it's gonna you to better
any if they have
that's really might want to use it i mean talk to provide the facts for
example to be saturate colours and if any of those we use
single that
so those changes all search about we don't need to do anything on top of
it to reduce power consumption for example
well it
it's is a magical you know
well there's a
there's a lot that went into
the patching for instance which is the main thing i guess that
think things like mention anything using that's are gonna take advantage of there's quite a
few things that need to come together to make that works not just having support
a that's textures that so
whole bunch of hassle to support other stuff like being able to expand from i
have a an embedded open G L E S device and it only supports non
power to text is a and there's tons of gpus arnie support that kind of
limited
actually a pretty so we have a high level textures in couple where we can
slide sections that when you have limited hardware so that you don't have to care
about that because the you i just want to draw rectangles with images that have
a battery sizes and not have to figure out how to slice that stuff up
to be of the map to some
limited hard website
it's quite a few things that
really make job
something like show
easier
does also help a lot with the stuff like video
i feel like that
well the code you see stuff is actually gonna be used but not the G
S T a line also data that so if you're using that then it would
be involved in video right it's also gonna and a bunch of things now they're
not the G S T could never do we always which it is
no we didn't really think you very carefully what we originally right
okay so especially that idea of being able to come should be have multiple components
contributed effects videos that at the moment there is no cooperative system to allow so
you
different G stream elements to contribute to that right
for video tape or something like cheese that might want to start a bunch of
affects together you at them i mean you'd be doing that by accumulating multiple ranges
to trying to you
those kinds affects a certain form of power efficiency point to be if you want
to have multiple of facts about the video did not make a big difference and
that's just the right way to do it it's just that without co operative system
for prime contracts
that's a difficult problem
so it is a nice one it's a recent development the logistic stuff we needed
to be have to support video effect
and rate
i think that'll be interesting to see if people to pick up
and