Name: The Browser as a Modular, Networked Video Synthesizer - Olivia Jack | JSConf Hawaii 2019
Uploaded: 2020-03-29T16:32:33.000Z
Duration: 27 min 17 s
Description: Thousands of YouTube videos with English-Chinese subtitles! Now you can learn to understand native speakers, expand your vocabulary, and improve your pronunciation...

Basic passive voice

Okay, so my talk is titled the browser as a modular, networked video synthesizer.

Um, and everything I'm going to talk about is online in the form of sort of a code editor and graphic surrender at this at this web.

Um, and this is sort of a tool I've been building.

But really, it's more of an investigation of three different things that might seem kind of separate collaboration in the creation of life visuals, possibilities of peer to peer technology on the Web and feedback loops, um, and has also become a platform for live coating visuals in JavaScript.

Um, so I'm gonna start with a reference from the 19 seventies.

This is the satellite arts projects by kick Galloway and Shay Rabinowitz.

And what they did was using using satellites in 1977 they streamed video from two different places and composited them on top of each other.

And what they said was that this marks the first time that the geographically dispersed Elektronik image was contextualized as a live immersive place where artists and others could convene and co create together on a scale that could be a CZ cultural inclusive of desire.

And so this is before the Internet existed.

This is just satellites, and they, uh, applied to get access to this this satellite and to send video from one place to another.

Um, and it's sort of this utopian idea that as soon as maybe if we just put two images on top of each other, we create this third place.

But it's this other place that some hybrid of these thes two places, Um and I want to stay here for a second because I think, um, we still use this idea of a place in talking a lot about the Web.

It's a website as if you're going somewhere or it's an address.

Just this idea that when you access the website, your you're going to a place.

Um, and it's sort of this analogy that doesn't necessarily have that much to do with the underlying technology's like, if I go to a website, my eyebrows are sending a message to a server somewhere.

That server sending messages back to me, I'm sending messages back, but it's not.

There's not that there's actually a place that I necessarily, um And then also this idea that it's live an immersive.

It was a good example of maybe we're just getting right now to this idea of a live immersive place on the web.

Um, so I wanted jumped a little too now, which is, um, and talk about Weber T C, which is a browser protocol for streaming video, audio and data directly between clients.

So this means that now in the browser, we can be a little less dependent on a centralized server.

Um, the server here is this pink thing or some some way that all of these clients can talk to each other.

But then Weber TC says that I can, as a client with a brother, directly send video, audio and data, um, to another person who's connected.

And in this diagram, I'm I'm sure a lot of people have seen it before, but but says maybe with this cut these kinds of decentralized protocols, we Congar go away from the left hand image which is more essential eyes network and go towards the right hand image, which is more of a decentralized network were where lots of different people can connect and exchange information in different ways.

I think being built on these this type of protocol iCal, for example, Webb Thorne, speaker browser, the dot protocol, um, and stuff like that.

And I just kind of got interested in this protocol and started doing experiments sending video, balancing video around the Internet.

Um, and as I was doing that, I sort of started to feel like this.

This thing on the rate of, uh, this network looked a little something like this thing, which is another reference from the seventies because percent reason, a lot of my references are from the seventies, huh?

Uh, but anyway, so what this is is an analog video synthesizer.

Um, and and basically it's a bunch of boxes.

Some boxes create a signal by the time that signal gets, so the screen on the right, the pattern of the signal creates a pattern, and the boxes are designed in such a way that many of the boxes can receive a signal and transform it.

And then So the order in which you connect the boxes will affect what comes out on the the final screen.

And this is much similar in analog modular synthesis for audio as well.

And so I thought, What if we kind of treat this thing?

This thing kind of like like this thing instead of thinking of of a website as actually a place.

What if we think about it as a bouncing signal that we can play around with?

Um And so So I started experimenting with this idea about 2.5 years ago.

I'm using Weber to see to send a video signal between various browser windows.

So here I'm sending this in this video to this window and compositing them with each other.

And then I'm sending this video in this video here and compositing them with each other over here.

Um, and these air all happened to be on the same computer to kind of show the idea, but they could be, you know, when one window could be in China and one some Well, one here.

Um, but this is I don't know why this was interesting to me.

It was just so let me explore this idea and because I didn't know what I wanted to do with it instead of actually making a user interface as I was experimenting with it, I just live coated in the consul kind of playing around with different things and and seeing what happened just with this sort of routing video streams and throughout the Internet.

Um okay, um and so Daniel Sand in the creator of that, uh, synthesizer that I was just showing key.

He said, Ah, the modules are designed to maximize the possibility of interconnection, thereby maximizing the number of possible modifications of the image.

And so the idea, his idea is that the more of those boxes that you have in the modular synthesizer, the more you can route signals may be, the more possibilities you have.

Uh, and so now I'm gonna move into the tool that I've been building to experiment with this this idea.

It's called Hydra, and it's at that link I showed earlier called hydra editor dot Glitch taught me, um and so it's designed to let people experiment with with what I was just showing you not just within by sending video, two different windows, but also just all within the same video with within the same window.

I mean, um and so I'm actually going to get rid of what's on the screen.

Um, and so the basic, um, unit within this is isn't oscillator.

Okay, so here we have an oscillator which eyes like one of those boxes in the modular synths world.

So here's ah connected the US later to an output.

Um, and you can think of in this case, this is a job script library that I've created for working with this model of sort of video synthesis, and you can think of each function as if it's one of those boxes that we're seeing in the modular synthesis world.

I can give it certain parameters like frequency, um, sink and also separate the colors somewhat.

Um, and then I can also start thio transform the signal so they can make it rotate.

Uh, I can make it into a kaleidoscope, for example.

Um, I can, um, pixelated it, um And so so here.

Each function is one of the boxes in the modular synthesizer.

And then the chaining of the functions acts like the patching cables people more from who are familiar with modular synthesizers.

Um, and this is this is interesting to me.

This is all running on Web D l and this chain of functions renders to a single shader.

Um, I think a lot of abstractions for dealing with graphics either talk about canvas like, which use a lot of drawing metaphors.

You draw shape, you draw another shape, Um, or that you sort of three d metaphors.

Like a lot of what we just saw, which is, um, a box lighting a scene.

Um and so here I am using this other metaphor, which is the the video synthesis and transforming a signal metaphor.

And so with this, if you want to repeat something instead of writing a loop and saying, I'm going to do it now 100 times it just apply a transformation.

So here, I could do Repeat, and so they could do it two times.

It's not more intensive for the process for the processor at all, because they're just transformations to the coordinates directly on the graphics card.

Um, and so that's all well and good, but this is just sort of a single chain of things.

I was really interested in more that mess of cables and how things were out into each other.

And so what I can do here is, um, I can have various various sort of output screens, and so, uh here and I can also use different sources.

Um uh, so they actually want to put this over there and that over here and said I have, um I have my to my two screens, and I can then also blend them together.

So, um, everyone up put called 00 and one called Owen.

So then I can now say I'm gonna use Ozio.

Subtitles ListPlay Video

The Browser as a Modular, Networked Video Synthesizer - Olivia Jack | JSConf Hawaii 2019

sort

process

pattern

delicate