us so thanks a lot for coming along is i'm right am reusing some slides
i gave
values for different presentation an internal sort of some it we have intel i hadn't
planned ahead to do this talk what they opportunity came up since vision a pistol
guess today to present this is
apologies about the branding and they didn't tell confidential as ensemble size you know that
okay so
my name's rubber private can i work for yet entails open source technology centre in
london
on a graphics engine enjoyed graphics for a long time
and just have the opportunity to work on a number of players in the graphics
to a linux ideas
one of the first companies i work so was imagination technologies right at the option
introduce silicon enabling
some about where and work through the open G L driver and some of the
kernel driver and that kind of low level work and
and i moved to
sort of a small company product and swear we'd it's discarded toolkit and gain some
experience with high level you are technologies and through into stress composites as and
or you can
the work in firefox
but so
as well as having this
passion for graphics
i've been fortunate to work in an office which is got cut indicated design is
visual design is interaction design as well as engine is the have been
involved in creating applications the design by these guys
and so i've seen the process by which we come up with ideas and we
create applications and so i hope they have some insight into where that process is
not as effective as i think it could be
and
so i'm here to talk about this project who rate which i started about
that you're going to be working on sort of part time which is this is
owed sort of markup
as you people missing from this and you know rubber's the is a guy i
work with maintaining project look always been helping with this time you must be a
he's on the graphics team at that in D C
i mean a is an intent have been working with that visit ago i could
chris cummings he's been helping out there as well actually we cows a visual design
it did this markup and help with the visuals of this project
the change rate has been helping out as well so
and so
i want to give a bit of background to why i started this project
and so the focus a good friend so i mean this the body aim of
this project is the to
to drive innovation in the space of user interface is we're looking to explore for
the use of the G P because
so
well actually just about because i i'm just gonna be a little bit rougher many
"'cause" i haven't run through this but so
during that experience of say five years of me working and you i technologies i
guess i've
built up this
kind of feeling that
you do we are completely you will not we were really wasting this gpu hardware
which is in our devices like mobile phones and a not tablets in a laptops
the state of you i technologies today we used a gpu for basically shifting tech
should rectangles around and so i've had this mounting feeling that is gonna be something
that we can do to
expose more creativity to the visual design braces an expose that gpu out with a
fully utilising the G P S as what i want to go through a bit
more
okay so and this and that through that experience of working with visual design is
what can we do to optimize the website from coming up with an idea to
create an application something just quickly recap state-of-the-art them
state-of-the-art
toolkits like G T K G T S waiting list plus the you pass and
even web technologies the that with familiar with that web okay
all of these are actually extremely similar technically speaking if you look at how they
work
and it's
they so they basically implement some form of the postscript or pdf rendering model is
the foundation of the vocabulary by which all of these technologies work
with and
so that vocabulary
in essence is these
three things it is just you have to D shapes you can fill them with
images there's two D shapes a basically just always rectangles i rounded rectangles if you
getting pretty fancy and you know theoretically you can use basic us and all that
fancy stuff but
who actually doesn't
you fill these two D shapes with images and you have text that's your vocabulary
i don't these are all built up you build you why up using the paint
is everything which just means that store the background and
will basically familiar with this a really a background press you met grant and then
your labels and foreground on top of that that's the model
and
you have all they we employ some form of seeing graph to organise the primitives
the which it's the components of the you why so you have a hierarchy with
the window the top and this is what defines
how you implement the paint is that within the stuff the top of the profits
of is the background
and the leaves a of the foreground just pretty simple it works to some extent
i mean the cornerstone of that model is the quality of those static images that
you get from your visual design that's
so these technologies
pretty much all i mean to engine is so that's part of the problem now
the come as an imperative A P I C python and such things
they have a markup language so
will familiar with H T M L with but there are things that you amount
as well
and it's
yes they all have this of sort of this contract and this promise that they
will as you develop these technologies you will to find a an interface for engineers
to work to and you must keep that's tables so
you know there's a button involved in maintaining a stable interface say that all of
your existing applications don't break as you of all that technology so is pretty darn
hard to think ahead of time what interface am i gonna be have to stick
to for the for the rest of time that i'm not gonna pay myself into
a corner as to how we can improve the flexibility of this
this technology visually speaking and enable more creativity for designers
it's that's difficult
so i sort of said this year this overwhelming failure of through this experience that
we could do so much more with the
more creative use of the gpu
so
this is spelling out of the be a the problem so we've got this the
first one is this limited vocabulary is the P D F postscript
rendering model it is like twenty five years so it's
and hasn't changed
so we've got the middleman toolkits this is the this sort of
liar where it's a means to an end these toolkit so just there for an
engineer to recreate the vision of the designer and this is sort of a language
barrier between that vision and the technology the designers tend to have very little insight
into the working of these technologies so the really obvious middleman and maybe we can
think about ways of not having quite the same
approach and avoid that middleman sort of
translation that we that we deal with today
and when we think about it from the designer's perspective as well
they're using tools like photoshop well in escape
and things like aftereffects
these are the intended for creating static images
and they have all kinds of fancy effects available to them by the end of
that you get a beta emission and we slice up and the rectangles and or
you are with that all the made for offline video
pricing like aftereffects and that uses things like ray tracing to come up with this
affects you know there's no way those kinds of effects are gonna run in real
time on a firing anytime see
and so
well these designers are actually trying to do is create interfaces that can run on
a results constrained device and be responsive to in the
and just you well within a power sort of constrained and it's
so they really don't at the moment have tools are actually geared towards that specific
problem so maybe we can think about
inventing new tools that right
improve that part
so
that is just the actual what fly
the very plastic sort of outline of how design goes of them i mean is
you have the special design a
creates sort of sketches out is vision in something like photoshop to indicate
and they had over the wall if you like to the engineer who has to
reinterpret that vision and translate that into and i pi a sort of fashion so
that the this station totally disconnected from the sort of visual creative side of things
that are opening up your reference manuals and implementing this and that implement is typically
on that desktop machine a laptop or so even if they are trying to create
something for fun you can initially priced at the stuff on your laptop
and once you kind of got it working then you're gonna started so that's integrate
this with appeal system to be able to cross compiler whatever it takes to get
it on T of fighting and then start playing around with that will how does
this before madison's response
and
at that point that way you're actually starting to test on the device
is a pretty high chance that you're gonna realise that this like some problem like
it doesn't the design doesn't fit the form factor like the designer which nist back
expected it to it doesn't respond as of as of quickly is it needs to
this like so many possibilities that could just not be quite right and you're gonna
have to go all the way back to square one where the designer has to
rethink the idea in the interaction model to be able to work with the technology
that they have
because the alternative is that you go back to engineering transfixed the technology but that's
the and you know an unknown quantity as to whether or not you're gonna be
have to fix a technology in time if you got the time so your options
are pretty slim about point to have to redesign things
and
another aspect of this is the ability to objectively measure certain
quantities of quality a that there are important for user interfaces
you a consumer expects that they're fine it's gonna respond in a timely manner when
they when they touch it lays we've around the you why their expected not to
redo that battery but i expect a certain level of visual quality and right now
we don't have the integration in a tooling and the
the processes by which to really make sure that there is
that we're on top of those quantities right through the whole design development process those
tend to just be a and often will not so much an article but they're
the end of the line
in the pipe line of
how we develop these things
so
i guess i kind of into the but some of the solutions to these as
i was going through the problems but
obviously the
one of the main things that we're interested in here is
throwing away that limited price good model and saying if we can explore way more
features of the gpu to
there's a lot of flexibility in a gpu what can we are not there and
give to the visual design of so allow them to just be more creative
the model of the price good model is actually really difficult to implement efficiently on
a gpu just given the way the gpus is kind of difficult to avoid the
over draw in the price good model because you going from background or foreground and
that's the model but you build up to see your it's hard to avoid colouring
in all of those redundant fragments and pixels behind the stuff environment of it that's
just quite a tricky problem
and so as you can probably guess from the screenshot of the beginning of this
talk the a big aspects
to how i've sort of see as being able to so many of those problems
is to develop a spec tooling
which is what the rig projects as is looking at doing
so what we're doing is we're creating we were actually
trying to learn from how game developers work here because
game developers face so many of the same problems that we face as you i
developers except that they have
done a much better job of taking advantage of gpu hardware than we have they
have to create interface they have to create something that can run in real time
response that you know interactively with the user interface with the user and we need
to do the same kinds of things
and
the weighted various guys work is that they have a rendering engine
and they often build up a lot of this but tools to help them create
their wells create that they games
and
they have to work as well with design is an engine is and if you
look at projects like unity three D where they have really quite so you know
capable tooling to be able to connect
that whole works like between
game design is and the engine is the created this logic and you game itself
so we're looking to sort of reports their ideas as much possible "'cause" we can
build on their experience especially they call of that on a lot of work on
into researching algorithms the less is the really interesting visual things of the G P
A
so
we want to create technology which is the you are rendering engine that we can
share between something that's gonna deploy onto a device
the route to run a U I and this and that the tool itself is
gonna be using so the design process is actually constrained by the capabilities of the
technology
so the visual designer was and then
the beginning and specialised tools during the visual design process
we want them to also the really connecting that be issue does development stage with
the hardware that they want to design for so we have
the i what we have in implemented even the ability to have a network connection
between the design so and the device this
that your you you're targeting so that while you're working in the design so
it's easy play around with animations of things that's gonna immediately a date
what's on the device so you can pick it up and check what's the responsiveness
like what's the quality of this like say that you're connecting testing right so that
early prototyping stage which there's just no substitute for testing your ideas on real hardware
so we see that as a pretty compelling way of improving how works like
and optimising the time it takes to develop
applications
the
the last one is necessarily can it will be done with within the same so
what we need to develop tools
for measuring save the power usage while you're running you so you testing your you
why on the device are descended asking you wanna be able to measure so what's
the power usage what i'm doing certain things or measuring the latency by which
you when use what
how long does it take for the interface to stop responding to that so even
we're even thinking of things like will high-speed be at video cameras and a processing
recordings of that kind of thing
we need to think through that problem as well
so
i guess at this point time to just
maybe but
if it was to just jump in and ask questions especially now that i'm gonna
basically demo a demo a then
just raise and stuff so
and
there's i wasn't really expecting to be there and then i guess and i've been
hacking on this the chances that this is
so let's this gonna go through a bit in order to this area on
on the last year is
well so that we're aiming at work can asset you can work load here so
we are aiming for
this area is where you import say things that we created that about does it
could be images made by design is impaired e-shop we could be models made introduced
your max it could be logic was created by engine is something that needs to
be integrated could be audio and so these kinds of things
and is this the sort of search area so we can search the images of
videos or whatever
whatever we need and then
people there is assets into the this centre area which is where we actually build
up the user interface itself we say we can we get immediate visual feedback about
the scenes that we're trying to create
on the right hand side here which is public to the moment but this bottle
would be where we would expect a more tooling capabilities and it's put there so
you can to the trends in the expand into the space of the interface to
give more detailed information about tutoring and when you're not using very spending can sort
of shift out of the way
and then down button here this is where we would so
deal with controlling of the problem is actually it is the best
just as the actually a where this is some of the properties
for the one of the objects investing a this idea of entities in components and
entity is like something ins is essentially just relative to a parent in space and
it has a bunch of components that can be associated that adding more semantic information
to the object the more components year at the most semantic meaning that thing has
and
these red buttons here about being able to control the particular property so if you
click one of those and it brings the ads it that this area down here
which is cooled what is to control is the initially when we were first that
this is just but online management but that's in the process of evolving beyond just
key frame based animation at the moment this is this example is just built around
key frame animations and it's just a
a toy them or
having in the idea i guess in this case is a fire an application that
we've got notifications bubbling up and something that's more recently people problem and it's gonna
ten years excellency can see a bit
visual stuff
so you've got down the field of facts which shows the something better way look
soft it's as modelling out real well camera works
and we're also playing around with shattered mapping in this example which is way of
having real time that shot as much
and we've also got real time lighting so
and you have a standard a lot of the lights based on and if you
which is a
so the
is a light source
well see we're ripping off at the moment it's
very traditional with actually seeing daniel engines this is largely a as a practical
the this
the practicality because
we have to bring up a whole so it's
there's
we didn't have the time to do the research and you visual algorithms this one's
a well understood and they work well together
but we'd be looking to do research on that's the
if you a stylised ratio that would be which is not something you'd necessarily expect
and again but would make more sense in a you why
but even these ones i think that there is scope being
so for having a design and it's have to use that a so say to
know go overboard because this does give you a lot of power to
you know
create a you why that would be and in massive three D virtual kind of
the you could easily go for the problem is a three D environment that you're
modelling within and but
you don't have to go overboard
so let me just the same time show some of the effects that
that
this one this one doesn't have i have got assets that really geared towards this
but it's a
a technique that as well uses the G P a bit more than where
doing them i wanted to bump mapping
but it's
your meeting so that that's twenty ten should have that image but that represents a
modulation in the shape of the a surface so it makes so it represents a
high essentially and
and you can really see the when i click that the service that it's the
it changed visually but what's kind of interesting is when you select the light
and you change the direction of the light goes out
didn't last too
so that's just to stop i found nothing special about that and the bum not
the case with this was just generated by running it through a program that uses
essentially so profile to that says there where there's contrast in the image that they're
edges best it automatically generates
so idea the edges you're moving up there
so if you take the last of the questions again you see that this like
a detail in this every stick gets changed the changes we make a lot of
this that this is not some is craft it's
what it's about this but i think with a bit of have a visual design
and knowing what they wanted to G fan with us dedicated assets then this room
to make something just bit more dynamic an interesting that light source might be doesn't
have to be moving know every time you move your finger around it could look
a little bit over the top but maybe it moves is
as you sort of time between spaces on your fine
another there's another this one is a fine example of just using the G P
to style as something the
show is
is the video on this
just to show the videos just to show really that is something that can be
done in real time and this point is an is court appointed is in the
fact that and all this thing is
great something on the original images scale this write up so you can see what's
going on
so as well
snapped and things to be have to
to control
and
we can
stylised something and this point is an effect
i guess used quite a bit in static images and prices of things
but we can do this dynamically and depending on the lightness of the pixel is
a it within the image then we're changing the size of some geometry to
to just create is a like nice looking at back to the
might want to have this in on a background of some aspect of your hard
merrier of a of a device interface is something that again have to design either
i get to leave it out to them but i'd like to be up to
provide them tools and freedom and but constraint as well two things i know to
run efficiently on the G P and it's almost cheaper to run this video effects
then it would be to actually show you the video normally just because in this
case okay ish
there will be in this case
a the i only points where we do like colour space conversion that why U
V to rgb of the double points which we are showing a secretary doing in
some ways less work than you would do to sure the emission and that's kind
of the but
so that's kind of the state of the two of them i expect i actually
they way in which the design so to next to the slave device actually which
nice
so this represents a device sitting on your desk which might be if i you
know say
well i it i don't have anything that's got this like running on a real
fine but this is the that we had an hundred back and that we created
a while ago and but it's like some updating also
so that there's connecting this
design to that device
celebrate from civilisation
so if i scrub through this animation then you see we have the effects are
enabled on this one well the not in the editing area so you can get
a feel for what it's like on a device and in here
but then
here if i gotta say to the end of the time line and grab this
change the animation at the end
about my device so anyway immediately outdated and it's see going to in czech what's
the timing of that change the key frame it's right into you constantly you're happy
with the work that while seeing it for real on the actual
and this done using of all here at the moment so ago zero come so
that it can just discover the device
this is possible
okay so that's basically will content about the moment to be have to introduce great
but be happy to take any questions and to
okay thanks to
question to you
the point to do so i want to run right well actually is the that's
not just i if
save for many with the times fragment pricing in fact expressing in terms of gpu
something like the future as well so often video effects i just done as fragment
programs like say you are doing just colour rising or you have to de saturate
or contrast or brightness is kind of thing you just running over every single bit
pixel in the image in this case where actually we have geometry for the cells
if you like in that affect
and
it starts off as basically a tiny unit rectangle for each cell which texture maps
a circle and when it reads through the videos data i need something on these
course great points so i really does like that why you be colour space compression
on a specific points looks at the lightness of those and then depending on the
like this is essentially scaling out that geometry so i've rule you're doing way less
fragment processing then you would often differ a bit effect a and you're moving a
bit to the you know the but there was really interesting to what i'm what
is you some that's in the tool designers can use
so someone to explore
i was so you wouldn't be able to see yes is so what are the
particular a design decisions made a is the it doesn't really doesn't really scale to
try to come up with them really modularising rendering system like at the extreme where
we have class to well i mean it's changed now but we you know with
to use to be able to have a so that
every single actor could write all kinds of in narrative card to do alighted roaring
and it's very flexible but if you want to optimize how use a gpu you
tend to have to do really classy with things and make various assumptions and it
works a lot better if you just make that piece of rendering card a lot
more monolithic so enrage it's much more sort of like a game engine monolithic rendering
engine and if you wanna extend a then that
the then you basically
the engine is to have this specialised experience to know how to program a gpu
efficiently they should come and should be every project and at their affects another features
that because the number of people that have specialising gpu programming
is barry lighter in terms of the number of people the ones right applications so
we can expect application developers to be writing G S L and we're you know
sort of shade ascends integrating their rendering ideas it a into a modular system and
also get the performance we will amount is that
so a couple questions the first is
what do you actually sending to the devices it just is just display list basically
with animation descriptions and i didn't have the network to device but well the whole
the rendering engine itself is running on a device or the description of the scene
which is we have this seen trough of
entities and entities just
there they just have a position in three D which can be write a sin
transform relative to a parent and potentially we could be sending multiple would be seeing
grass and each component has each entity has a bunch of components which say ads
the idea that you have material which says how should be lit or some sort
of video source image source it could be some logic associated with that entity that
triggers when you have input all this what kind of high level
sort of semantic description it's what's gonna get sent over the wire to the device
and it's gonna like that
second of which in the my second question you talked about it kind of like
already against are they created a their scene with reagan so like a game engine
so when it comes time to actually like should the application of a shipping the
maybe so what you know where they shipping the all the assets and then this
description it's interpreted
well that's because the way the than the network writable has to be able to
serialise everything that relates to be the description of the scene including bss themselves and
we also use this so we using part school buses which quite widely used way
of civilised later and they give you a nice markup for describing messages for stabilising
data and we use the same particle we use the same mechanism for being a
to stabilise over the network as we do for stabilising to this when we is
at the moment when we see realised that this we're keeping references to external assets
but if you're gonna make it for distribution on device then we would also be
see realising bss themselves in the same way that we do for the network and
you would just have this one top rate file which you just
and put on the device along with the actual read
sort of engine itself built in three ways the i the bills with all of
the editing capabilities around it or it builds
in as a slave the sort of it as a slave so the one on
a device which has that network transparency or just
for deployment of device it doesn't need to network transparency doesn't need any day editing
capabilities it's quite got basically no dependencies exact G L at that point and
huge ship that device might plus your top rate file which has everything sort of
sailors together
so one final question have you considered a
a web device so i probably but it actually say this so we we're interested
in supporting a bunch of platforms with this and that includes web development in fact
so this way we have got the fold them are working but i'm the i
at least we want to do the proof of concept is
see if it's feasible and i'm kind of optimistic that is because we have the
rendering side working with web G L and so you know this depends on G
L to do to have access to the G P we trying to projects the
gpu the only
weighted we can do that in the context of the web is web geo and
it looks like that's gonna basically be
across all major browsers that is soon including i and so
that's
that's where we're looking to go
so you mentioned that the friendly model we should try to not a bill with
like you know the we that anymore so you have a specific a specific examples
of how we
so apply principles that re well probably uses to get rid of the P D
F really model like well run vision even this initial example but the way that
you build this particular scene out it's got intermediate rent is where you have to
first determine that if this geometry up before you're able to come up with the
blurring this based on that distance and the there are multiple stages involved in coming
up with that final is so you have to do it is a it is
a step where you look from the perspective of the like for instance just say
well from the lights perspective what geometry can i see said i can determine what
geometries in shadow and you know this there's a whole bunch more involved in that
model of rendering which is completely different
the postscript model of you know you just draw your background you draw other stuff
on top of it in the middle and you draw your labels and stuff the
is just a that's a completely different model you couldn't achieve
this kind of thing with the postscript model in real time
but is the other things like i'd be interested it's at least exploring some of
the stylised to painterly rendering or so where do we can make something look a
bit like it was painted with oil paints or the like it was sketch just
something aesthetically
you know interesting there is a
that would that would take your completely different approaches to rendering as well then even
what i just described there
and so these are the really quite far removed from that
painter's algorithm hands and the vocabulary of two D shapes text and images
is that
that makes sense
i'm thinking of something like the broadsword browser for example yes you know very complex
you have to support a lot of different techniques for throwing a so probably we
could not completely get rid of all P D F and the remote but maybe
some of the principles that you a scribe know could be applied
so probably you have some specific you know experience with their potentially be way so
you could
other ad rendering a new a rendering model capabilities within the browser but it works
i to be sort of
sandbox to some region and then you get this new model to use in a
certain area it would be challenging to fit the postscript model within a different model
in that
this in these in a lot of these cases you're using that to determine a
something is behind something else so you have to other
trying to fight the idea of the postscript tiring in terms of shifting things that
you might be able to sort of rhetoric that's on this about stuff into the
and
a some browsers which has
but
maybe we can chat afterwards of that's different is if you interested
go thanks
so