say hi everybody thanks a lot for coming on to my talk about global so i'm robert right guy work for intel open source technology centre on the kind maintainer of the whole project so for those are not familiar with probably it's other consoles gpu graphics library originally created for the cluster projects they are these days as a standalone project it was created to deal with the possibility across different versions of opengl so course opengl yes opengl and even experimentally a bit these days with the web you know it's is so dealing with portability across operating systems one of the differences with some other graphics if you refuse are base is that initially when we created it we were interested in supporting more multiple sort of orthogonal components being of the contribute to the graphics rendering in a way that didn't trampling each other's us take tracking so in a still come hoping to go through first a run through of how we submit and process geometry with a gpu and when the something out and i want to then give an overview of the utilities in the couple library that try to make it really practical for writing graphics applications be on just access to the G P your hardware and then talk about some of the specific things we've been developing more recently just say that people get some insight to how the project is actually developing so first so wanna talk about basically the lowest level api even in cover which most closely matches have a gpu process is data i'm kind of personally interested in trying to expose more people to understand have a gpu itself works especially in the context of user interface development where i think we try to be on the utilise the flexibility of this hardware and the state-of-the-art for you are is today this is quite basic in comparison to you can see to sort of game developers and how successful they are using this hardware with many of this and that kind of constraints is us with interactivity and real time graphics so anyway the first thing that any applications gonna have to do they're gonna use google to draw something are with the G here is great and of the cobble context this is gonna just you could cool context news is gonna create your sandbox for all the data the call was gonna need to track for your application so then we're gonna need to create a so it's a destination for what we draw remove are rendering this is called a frame buffer a frame buffer is a collection of to debug is the but we care the most about is the colour buffer and that's where we're gonna store the results visually of what we're drawing but a frame of mind also contain other ancillary about this so that are is for accumulating the depth of the geometry that is used during this that we have to see so you know these guys in probably create a frame buffer you can you can great and on screen frame buffer that would be visual to a visible to use of coding got one screen you with the size alternative you would create an offscreen frame buffer by creating a text you have with a given size and then you would rat that texture with as an offscreen frame buffer so if we have a destination that's where we're gonna store results then we need some data that we're gonna process with the G P A and how does a gpu except data so and gpus also actually the is so the thirty forms of data so we i think so the in cover we support lighting image data have for sorry in the order and image it's a as a form of a so that we would expect we have person a gpu be on search german tree and it's a in probably support lighting textures on the from finals from afar name or from about four with a given size that and by sending data on that texture us one is being up to images into a larger at this of images a and it's often useful to be able to how can we just again as a high level protection the way you often it is a compact and to be have the packs more images into a larger images to avoid state changes and another form of textures the gpus process of pictures great from when your system resources such as X eleven expects or while and i'm sorry the older this okay it's just that the left and right click it doesn't very this is a real andreas breaks the slide down to the sections which of vertical sections like this pressing that right skip saver pretty big chunk that so which so about a so this'll spree this slightly better slide talk a week or two meeting date to the G P so depressing we need we have to do is the actual geometry into the G P A so first we need to actually understand what they to the G P processes and that is geometry which is test selected from triangles is the primary form in which we price estate on the G P A so you to understand you have the how we describe astrology the next thing to understand that's a this up about that is the stuff like just jump ahead at and sorry about that but we're gonna fast go through explaining how the geometries this but apologies and so the way we describe it geometries might tell is german treaters trying plated from attempt to assign different triangles and there's triangles themselves made of that but to cease and that is that sees are themselves components of what are called attributes so examples of attributes he would be so positions for each vertex in a triangle so you would have a position which could be a coordinate two D or three D coordinate reach but text in that geometry or it might be a colour for each vertex in the geometry or it might be i a texture coordinate that would could be at to detect record there it is the mapping a specific subregion of an image that is a don't you there's a lot of flexibility in the gpus to what data you can associate with about season your geometry and these are just very the examples but it's you can have all of the cost and data so i see the outcome actually but they don't you can also have many different apologies so tip about apologies would be that you have a triangle list which is where you would have so i sets of three but the sees which represent completely disjoint and disconnected triangles or you might have a strip of triangles and that would be you divide your first set of three but sees feel first triangle and then one additional verdict is gonna connect to the last two that sees of your script to make one more triangle and so one more vertex connects the last event these answers a more data efficient way of describing some geometry than with the less if that's applicable to your which on your trying to describe and last example would be a band where you just have a single but clicks in the centre and then the rest of the triangles gonna spend a lot of that and that's and one so how do we actually described estate using the called I P I so the first thing we're gonna do is describe decide what kind of data we want to associate with the each of those matrices there is some different strategies as to how you present estate to the G P so i'm just gonna give you one very typical like how we would do this so we decide that we wanna have a position the colour and texture gordon associated with each vertex in our geometry we're gonna create a since even greater about six am i-vector extractor in this case and it has a position it has two components of the position X and Y coordinates into D and there of type fly we're gonna have full components of the colour but we implement out for a and each of those is a by a number gonna have two components protect recording it and those are touch like nist one and this but what about an this is just that we've taken on my vertex struck which is a vertex i'm with declaring an array of these and we're gonna plot out we've got three about sees for one triangle and with plotting out the positions of the but sees giving each method the colour and where specifying the texture coordinates you so we nine to up like that data to the gpus able to read and prices that and that's just a case of uploading into a cover that should be but we do that by coding a battery but when you repulsed in the size of the data that we want to apply time we pass in the actual point to the data and that's gonna give us a reference to about the next thing is that we're going to have to describe the G P or how is it that it can what we got data needs to know that i have to that data so it's right three the but sees to process it and so if we reach out to be we need to create a cover that should be and we go global attribute new and we're gonna pass in a reference to the buffer which is a date there are actually so we do we declared and we're gonna give each attribute and nine we're gonna specify what is this be and the number of bytes that we need to step to be have to move from the best the text of the second but sex and so on and then we're gonna need to specify what's the offset from the beginning of the buffer whether gpus gonna start reading for that should be here to process that data lastly we need to specify the number of compared to data have you have to take like two components so assuming we do the same thing for the colour and texture coordinates we have for batch we recovered attributes now we can now define one or more primitives the referring to that should be right so so at this point this is where we specify the actual topology that we want to that we wanna gpu to use to interpret that data so lists triangle friends trips or whatever so with the funny primitive here way because it's just a single triangle that apologies pretty meaningless so we specify that there are three but sees the belong to this specific primitive and then we specifying the actual attributes the belong to this primitive and in this case we using this of our function takes a know to make the list of that should be says so the other thing that gpus comprises besides the john geometric data that i just described that is that they can also still image data as textures citizen where we were a bit outflow of their own and i started going through how in cover we can like text data so we have we can let them from files we can load a texture just as an empty but and fill out from the C P with the point to we can also create these textures which are cool data structures like an example of a high level texture the pose able there are numbered if i don't use but it's sometimes useful for this band of packing images one texture to avoid like a changes for you are use cases and might these pictures from when the system resources what is and so we probably talk we put off right but what we're gonna store results with what our data which is these triangles and images as textures so now we need to figure out how to describe the pipeline by which this is all gonna be processed so i'm gonna give a really sky high level view of any it example of how you can configure gpu pipeline to process data of modern T is pretty don flexible and they're a number of stages a cut optional state is really but that i'm not representativeness example but it is still a very typical a common example is for is this technology they're gonna start with our geometry we coming in on the left and we're gonna bust going to this but expressing stage in this stage we're gonna transform a german three we're gonna scale it rotated translated we're gonna projected so that points at a button away from the origin are gonna attenuate towards vanishing point to the realistic and at the end of the stage all about german tree is gonna be transformed into the coordinate space about frame but and it's conceptually a range within a scene of other geometry bands the next stage after that is the frogman processing stage in the G P is gonna go through each of the triangles in our geometry and it's gonna i would like that with the pixel grid about frame buffer in it's gonna consider for each pixel where the centre of the what's called a fragment where that is within the bounds of that triangle then it's gonna run a program to basically calculate colour for that fragment at this point when you're calculating not colour you can but that's of those textures the we floated and also the coolness of each triangle where we have those attribute values coming in there is can be interpolated say that for every intermediate fragment network processing we have interpolated attribute values to contradict program so in the final step we take all of those fragments and we know need to humanise there is onto the existing contents of the frame buffer to build up a whole scene and it this point we might also be reading back insisting colours in a frame buffer to be able to say if the effect of transparency by combining in coming destination ones and so the buttons wasting the from present processing state is there is programmable stage is very flexible the planning stages tends to what is a much more limited functionality done and fix function of us still a can lay down do that stuff and and the well in which the actually the hardware prices this is very paralysed there is if you have a primitive without one for triangles coming and there's not really any bordering guarantees as to have a gpus gonna price essays triangles and it's gonna be doing lots of geometry the same time and similar with a fragment processing the order exact order in which is gonna priceless fact is not something you get to guarantee and it's very paralysed that you will be doing sorry so kind of have a gpu prices to state a high level and i don't have the time to explain lots of examples of using the cover if you have to describe a variations on configuring E D U but just to introduce you to the couple plot line object which represents that whole processing part by the G P A and we can create a peaceful pipeline by conical pipeline you and to just give one example thing of for figuring it where we are introducing a texture into that pricing but we're gonna associated texture with what's known as an idea and this idea of layers use during the fragment pricing stage where they can be combined and us associating attention with layers there of that so basically now we have all of the bits that it takes to actually draw something a process similar gpu i'll just mention frame buffer of data a primitive we got a processing pipeline we can call primitive draw busting the reference to that data that frame buffer and is that the choice of five line and so that's basically that's basically it but introducing datum pricing G P A and it's it is to distill as much as i got time to be able to i give you so the next thing i wanna talk about the utilities at all the whites that go beyond just that lower level access to the gpu hardware that trying to make it actually really practical to get stuff done is just basic things a little mice all graphical applications need and gonna start off by introducing the math utilities that come with bubble we have matrices api matrices of the way that we deal with describing transformations of geometry in combining those transformations just essential stuff really matrix not build on top of that capability and kate to the number of applications that want to organise that the geometry into seeing profs so it's very common that you want to basically start these transformations together is interested in seeing trough pushing them onto the stack "'em" popping them as you move around the scene rough and this can say you re accumulating lots of transformations about this is the on changing i frequency and that is it just you know use of all kinds of things the next can be to have a you know it's on and dealing with that is rotations there are several rotation types that we have been a couple user an example of a compact representation rotation comprised of three orthogonal rotations and quaternions or another format it's an encoding of an axis workstation and they have the nice property being able to interpolate really smoothly between the have the pros and cons and but especially with quaternions they're pretty some pretty fancy matt's behind the it's not every time you need and colour utilities so being able to convert from rgb take you saturation like two spaces other which i just snippets this is the way that we experienced the programmable parts of the program will parts of the gpu this this design say that you can have multiple components contribute into the i roll processing of geometry and it so cooperating within one framework so the way this works is that we have this idea that points in apart line so examples would be vertex processing stage where you wanna be able to make some it's you want to build on the people processing of probable it but introduce some additional transformations for example you can provide a small snippet of code which is using the G L of cell shading language to insert into one of these have points in a can either replace that whole section replace any people processing or complete and or pen so that it can cooperate without science and similarly for fractal processing like and set to and that supplied to and looking up textures and there are places as well clipping lots of applications once we have to restrict the range where they want to restrict where the influence of drawing within the frame buffer to a certain to D regions and so if in cover you can click to ask align rectangles transform rectangles this interactive a battery primitives to D pass describe with fancy is awesome lines and similar to the matrix that we consider the you have seen rockies cases where you want to be able to start these on top of each other is interesting trough and effectively region is the intersection of all of that is the company push regions rectangle patching especially when it comes to user interface is a currently very dependent on using tech should rectangles as the so building block of most interfaces and because gpus are not really very effective a practising very tiny primitives one of time just to send a couple of triangles for what within separate draw close is is not and make good utilisation of the of the G your hardware so it with this there is the if you well within certain limitations i for instance with if you little your textures into atlas textures so they can all be referenced as part of one two like one month one primitive and if you limit yourself to certain types of clipping then in a lot of cases actually couple can batch almost like whole scenes of rectangles into a single goal cools it's quite useful so don't have a bunch of optional sup libraries that come with problem as well i mean for instance we have come up anger which enables us to rend text laid out with that anger make you know i we have hope a which provide a test like to have a teapot so describing shakes with bessie is lines and curves and being able to get a retains object representing that estimation of that and being able to redraw that without incurring repeated incurring the cost a test selecting that geometry how would you nist two thousand is to easily integrate a just regular gas to carried into the a couple application and managing the state between nice to it you know it's i would use T is a recent addition which i i'm speedier integration so couple based on the G stream a framework so now ago three some of the much more recent things that we've been developing just say you know idea of where the focus is a and where it's going so that's the yes i thanks to find a at the beginning of this year we introduced a couple frame info right you know i and it's this is about being up to get asynchronously extra information about frame is completed so for instance when we went to remove also possible frame of the impostor to be displayed and it takes some time before that can post a response and gets that frame actually displayed on to the monitor and we'd like to know too well what's the time stamp when that really hit the display and get that but we have location so that it can be a bit smarter about our rest is animations based on the prediction or when that frame is really gonna hit the physical display see you have these i think thanks to kind of stay in for contributing offence compensate pi two call this allows us to insert marcus into the come on stream that was sending to the G P A so that we can have a bunch of what that we're interested in knowing when the G is finish processing this way since when that work is finished then we can get a call back into the application to be able to i respond to that and the use case they had this was for video integration where they wanted to be have to make sure that but the gpu had finish any something the need to do a certain textures before was handed back to the video decoder library to avoid conflict thanks to i mean a man a lot of one of the interns so we've been working with recently and saxony a lot that's the supporting i'll at this work for G S T on speedier integration to couple and it's it works particular well with to pay particular attention to making sure that it works nicely with the state i the eyes and with multi texturing side it is a lot of freedom to i will all kinds of your own custom affects and things that will be involved in a rendering of a video frame so if you wanted to do from processing like be saturating contrast brightness that kind of thing and trade is you don't wanna have intimated rents we listing so you need a cooperative system to allow you to combine these effects one and as you have the use cases this was for project called break sure mention that so what you have is another thing that were exploring at the moment there's a project that where this using couple that i'm working on who rate which is i you are rendering engine and provides interactive tooling for designers for improving the web design where were exploring more advanced uses of the G P are in the context if you i development and it's we're interested in supporting a number of homes and even we're interested in exploring supporting web applications it's an expiration price my remember where i will be possible and okay so what's next so something that i always stripe in the way that in terms of maintaining the project and driving a project for this is ensuring that the feature what we develop it's really driven by the projects work the requirements of projects are actually using so we don't wanna be planning architecture astronauts just coming up with fun stuff like that we wanna actually have a reason why we develop any particular feature so the ounce really this question is to consider the people using the cover like my the for new uses a couple will take it and you directions so some of the current users a couple that implements it we have a lot to obviously are the do the first user couple and it's even today the that's a influencing certain things for instance the high dpi stuff so i guess they will need to make some small changes to the called tango at are to consider a the size of clips that we tool but there is a special use case of a couple especially when we consider the recent web two wraps name show that way to support and i'm show and that the bonds a bunch of the feature well as well in kabul well who is not something i'm working on activity at the moment but it so i wanted to mention it just because i did put quite a bit of work into this a few years ago and it was an interesting project it really do that should drive quite a lot of thank you think over the time and this was adding a gpu acceleration back into card was taking a different approach to carry G L and are generally have said making some different tradeoffs but it did some interesting things to do if retaining the translations of parts and trying to be a little bit okay well about using the fix function possibly gpu for gradients and the be a bit more efficient i would say much about you best "'cause" i guess not many people here about that but you know robert's writer back and for E best the i this is like a couple years ago we played around this is yet not really got serious use case but it was interesting the time because and there's actually some slightly nifty things that you best dates to do with matching actually which we were interested "'cause" so and so a much more recent projects that working on this week which i mentioned just briefly it's a okay and i don't hide for this project to be quite a big driver a couple going forward actually that mission here is to really explore in the context of user interface development how we can use more features of the G U and how we can also optimized the web fly between design is prototyping but at the moment without many constraints this to the technology used in the end and creating interactive tools the hell optimize are what are even connecting we to sort of devices i network and such things and so right about that big driver so those of the project striving that's night it sort of update for the projects and so i'm entrance it's in it if anyone's interested in this kind of thing graphics processing and it's especially a why and this audience in the context of user interface development i'd love to invite people to learn more about the G P A and contribute to call i've either directly or through downstream projects and creating a more that downstream projects and we we called wants so join us about join the mailing list get involved in discussions during this and i have see we created a channel for call dedicate the corpus the other day about time really just the program trying to send a message on again off together and say for us on better if you just want to i project and there's a website now find what you what you will we we're gonna probably try and i mean it but the labour office i dare easy which will time nine what to make it as easy as possible so that i'm sorry any questions gonna task the this building what this piece to me well are you use the even if you're using name show the patching that it provides a would make it more question it's gonna you to better any if they have that's really might want to use it i mean talk to provide the facts for example to be saturate colours and if any of those we use single that so those changes all search about we don't need to do anything on top of it to reduce power consumption for example well it it's is a magical you know well there's a there's a lot that went into the patching for instance which is the main thing i guess that think things like mention anything using that's are gonna take advantage of there's quite a few things that need to come together to make that works not just having support a that's textures that so whole bunch of hassle to support other stuff like being able to expand from i have a an embedded open G L E S device and it only supports non power to text is a and there's tons of gpus arnie support that kind of limited actually a pretty so we have a high level textures in couple where we can slide sections that when you have limited hardware so that you don't have to care about that because the you i just want to draw rectangles with images that have a battery sizes and not have to figure out how to slice that stuff up to be of the map to some limited hard website it's quite a few things that really make job something like show easier does also help a lot with the stuff like video i feel like that well the code you see stuff is actually gonna be used but not the G S T a line also data that so if you're using that then it would be involved in video right it's also gonna and a bunch of things now they're not the G S T could never do we always which it is no we didn't really think you very carefully what we originally right okay so especially that idea of being able to come should be have multiple components contributed effects videos that at the moment there is no cooperative system to allow so you different G stream elements to contribute to that right for video tape or something like cheese that might want to start a bunch of affects together you at them i mean you'd be doing that by accumulating multiple ranges to trying to you those kinds affects a certain form of power efficiency point to be if you want to have multiple of facts about the video did not make a big difference and that's just the right way to do it it's just that without co operative system for prime contracts that's a difficult problem so it is a nice one it's a recent development the logistic stuff we needed to be have to support video effect and rate i think that'll be interesting to see if people to pick up and