criticalartware



	/ (.jkC) Josh Kit Clayton / ======================================================================== / applat vrsn 2K6 / ======================================================================== / Josh Kit Clayton is a musician, developer and computer programmer who develops Jitter for Max/MSP. Jitter "extends the Max/MSP programming environment to support realtime manipulation of video, 3D graphics and other data sets... Jitter abstracts all data as multidimensional matrices, so objects that process images can also process audio, volumetric data, 3d vertices, or any numerical information you can get into the computer." (0) As a musician, Joshua Kit Clayton moved towards including realtime video in performances of his musics, musics that can be variously categorized as electronic, laptop [+/or] glitch. recording under the name Kit Clayton, Clayton tours internationally + has released music through Orthlorng Musork (the label he owned + operated) as well as through Cycling '74 (the company for whom Clayton works developing Jitter). Jitter's introduction in 2003 contributed significantly to the expansion of cinematic approaches to realtime audio + video, especially appealing to developers already familiar with Cycling '74's software product Max/MSP. - jonCates 0: data.src: title: Jitter Overview dvr: Cycling '74 uri: http://www.cycling74.com / ============================Josh Kit Clayton============================= / / JOSH_KIT_CLAYTON.INT / / NTRO / Technology is power, and we have complete mastery over our environment through technology, and this is leading toward a better human experience. That statement I have serious problems, or issues with. What I'm interested in doing is making software that other people can make software that is artistically driven with. I'm not interested in a mechanical relationship between sound and video, even though the software I write is often used to that end. Once you're in a numerical space, you can apply any type of logic you want. / LOAD / Well, I don't know that I necessarily view what I do programming-wise as an art-form. I mean, it's certainly a trade or a skill, or you could see it as an artform perhaps, but there's no fundamental artistic concept in the software I produce. What I'm interested in doing is making software that other people can make software that is artistically driven with. So in that essence, I'm very interested in giving birth to that type of software, artware, whatever you guys call it. But the software we make, Max/MSP and Jitter, is not in itself artware. That in and of itself presents problems for us, because we have to figure out: "okay, what is an artist going to need that is somehow different than someone that's accessing databases and stuff like that. So, I mean, we have to think about artistic concerns and needs at the same time that we're developing this stuff. The nice thing is that with our software, there's a pretty good dialogue between the users and the developers, and so we don't really have to extrapolate too much. And many of the developers are artists or musicians in their private lives, so that also helps drive what we're going to develop. But the fact that we can just take user comments—they tell us what they need, they tell us what they want, and you know, sometimes we ignore them, sometimes we satisfy those requests, and that makes it easier. But there are times where we have to evaluate, you know, what do we think is more important for people in general. We try not to impose any either aesthetic or artistic dogma to the software that would prevent anyone from realizing what they want to do, but there are obviously inherent limitations in every application developed. So there are certain things people can't do. / JIT / That arose out of my frustration with live electronic music in my own performance, and problems that arose from performing and presenting a theater that was simply a consequence of the music production, not necessarily what I wanted* to be presenting. So, for example, sitting behind a laptop or sitting behind electronic equipment, presented this theater that I didn't want to present, but it just happened because that what people organizing shows want; you, on-stage, and they put you there. So then, after a while, I stopped performing on-stage, because I didn't want to present that. Then it was the theater of emptiness, right? So there's this empty stage, which people still stare at, which is really cute. But then, I started working on visual processing to try to have some visual component. Often times what's produced is what's generally referred to as "visual music," or something like this. While I like visual music a lot, there's a lot of criticism of some of the art that happens, because traditionally people are less receptive to visual music in a performance context than they are to audio music. I'm not exactly sure what the reasons are for that. /* N-D / So in order to discuss what Jitter is, it would be good to identify what Max/MSP is. Max/MSP has traditionally been a MIDI and audio programming environment. It started out with just Max, which is the core engine and event processing environment, where users could connect objects in a graphical plane similar to flowcharts, where you would have Object A that could communicate to Object B by drawing a line of connection, much like patchboards that an operator would use in the old patchbay phone circuits. That was originally developed by Miller Puckette at IRCAM in France, which is a music and acoustic research institute. In 1997, my boss ported the Pd—well, so, the older Max/FTS code chad been ported to a native processor by Miller Puckette, and that had been called Pd, "Pure data." And then our boss ported the original Pd audio kernel to Max. That added this native audio processing component to the environment. Then, Jitter was just another extension of that, so not processing events, not processing a one-dimensional timestream. It was extending this processing to higher dimensional datasets. So one of the most common ones that is used is video, or image based processing, and there you have a two-dimensional frame—you know, a single sample is two dimensions, and you might process a stream of those samples. Also, it's possible to process single frames or matrices of arbitrary dimensions, so not just an image that has an X and Y component; you could process a voxel, or a volumetric sampling of data that has an X, a Y, and a Z component. So you could processes cubes of data, or hypercubes of data, and that data can be of arbitrary dimension both spatially and in certain terms, chromatically is often used. What that is more explicitly is you can have arbitrary dimension over the range and domain of the data which is being sampled. In our system, we also use it for processing text streams; chunks of text, we can also process particle systems, where elements of the matrix are your particles, with their life, their spatial position perhaps. Or 3D geometries—maybe a connected mesh that's implicit in the data organization of the matrix, or not; you know, you can have one matrix that specifies a set of vertices, another matrix that specifies the connectivity of the nodes... You can use it for anything—anything that lends itself to a, I don't know, 1D, 2D, 3D, nD representation of packed data. / DATA / Just as you have an environment like Flash, where there's this timeline based assumption, and everything... I mean, nowadays, you can circumvent that a lot easier through their scripting language, and various other means of making events happen, but when I first took a look at the software, I couldn't do anything with it—well, I wasn't willing* to do anything with it—because the timeline-based model was just completely antithetical to what I wanted to do. So, where you jump—you create these moments in time that have state; these are the nodes inside like a finite state automaton, and you start jumping between temporal moments to simulate this kind of finite state machine. There are many different ways to construct a dataflow environment. Even inside Max/MSP and Jitter, there are multiple models. So, MSP, for example, it's a synchronous model, where everything is tightly coupled to a clock device—the audio hardware—and all the data's flowing through the system "at once," as it were. I mean, it's sequentially programmed, but if you have an object with four connections coming in, it receives all four of those signals at the same time. Whereas in Max, it's an asynchronous model, so you could have four connections, and at arbitrary moments in time, each one of those input points to the object could be receiving data. Jitter, similarly, is an asynchronous model, and it also institutes all this stuff like "drop framing," that people need to do, in order... You know, in an expensive area of computation like video, it's very easy for the user to quickly come against the processor wall, or the memory bus wall, or whatever it is that's the bottleneck, but they want it to still keep working, and work effectively in realtime with drop-framing. And that's acceptable, typically, in a visual environment, because when there is a frozen image, we don't have a crack, or a digital glitch. I mean, it's visible, and you can see there's a glitch taking place, but it is more acceptable to our eye than a dropout is to our ear. So those assumptions are made. There are also assumptions made based on how data is converted to one or the other format. Say you have a heterogeneous data stream—what do I do if I have a ten by ten image and I want to mix that with a 400 by 400 image? How do you map that? So there are certain assumptions made to resample the ten by ten to the higher resolution, or downsample the higher to the lower, based on... I mean, this is getting really technical, but this is just to illustrate that there are assumptions that are made and these types of assumptions are real concerns to anyone who uses a piece of software that may or may not be driven by different inputs. In Max there's this "leftmost as being dominant" paradigm, so an object, a processing unit, can take input from multiple sources, and do certain things with it, let's say, simple example, crossfade between two video streams. Well, the leftmost input will drive the datatype, the resolution, and certain things like that. So yeah, there are all sorts of assumptions just in order to make the software work at all, and then you can get around those if you want. /* APPS / Even if you're only concerned with working in video or 3D graphics, the arbitrary nature of the design is useful. It's very data-neutral, and Max traditionally, while being a MIDI or audio environment, has focused on making it as neutral as possible, so that it can be used in a variety of contexts, so that once you have data in the system, it can mean anything. So you can convert MIDI notes to servo motor control voltages, so that somebody can program an installation that, based on musical notes, might have some kind of action, etc. It focuses on a numerical treatment of data, and once you're in a numerical space, you can apply any type of logic that you want—well, not any* type of logic, any numerical logic that you want to the datastream. Similarly, with Jitter, while it focuses largely on video and graphics, they're just numbers, and it tries to bring that to the user as easily as possible, but there are still marketing constraints, and the majority of people are using it for visual applications. There are a lot of people that, coming from a musical background, will get into Max/MSP, and then they start developing software, because it's a very easily extended language. Back in the day, it used to be easy to just program something simple, and you could do a lot with it. Now, in order to get anything to happen in a large-scale windowed environment, there's a huge amount of operating system glue that you need to get through. So with Max/MSP, you just write one little thing and you can run it in the application, so a lot of musicians have gotten into the programming side of things. Similarly, a lot of people that are coming from a musical background are now getting into the graphics world, through Jitter, which is maybe their first exposure. That's actually kind of one of the difficulties for us as far as making documentation and tutorials, because there's a whole wealth of knowledge is often assumed for the musical background that you don't have to go into in too much detail when you're presenting a music application. But when you have people that have no education in image processing, more than opening Photoshop, or whatever task they need to do—and especially video, dealing with issues of various international formats, or interlacing, and colorspace, and all sorts of things. It makes it difficult to try to present documentation for the software that's also an introductory course on video concepts. /* TECH / Maybe something that I'm personally thinking a lot about these days is a frustration with the way technology is presented; both by software companies, and by artists that make use of technology. The way it's presented is: "okay, technology is power, and we have complete mastery over our environment through technology, and this is leading toward a better human experience." That statement I have serious problems, or issues with. Not to say that technology doesn't provide certain realities that I would value higher than the alternative, for example, the successful birth rate is leaps and bounds over what it was hundreds of years ago, so technology does provide these advantages, but it is not this panacea of human fulfillment or something like that. There are still lots of issues, and there are still lots of problems that are not addressed at all with or through technology. It goes back to the relationship between the artist and the audience; I also don't like this traditional artist/audience that's based on virtuosity. I might appreciate a virtuoso, but the social structure that moves toward this power relationship between the virtuoso and the audience, or presenting an event or a piece of work that is monumental or exciting, to impress the audience, is really something I'm not interesting in doing at all. It fails to address certain human issues. I have a sort of inherent draw towards the unimportant or kind of the mundane, the everyday... Things that don't easily fit into the economic—like, I can't say why. They might be things where a transaction is taking place, but it's incidental, like, I don't know, this plant might look like the Virgin Mary wearing, you know, a Ronald McDonald wig, and I'll be compelled to that, but that transaction... that part of the plant is not trying to sell that to me. / VALS / I'm not interested in a mechanical relationship between sound and video, even though the software I write is often used to that end. I mean, probably the biggest problem it presents is: how do you value anything that doesn't fit into this formalist tradition or a classical notion of beauty or aesthetics. So that's difficult. So people come up with all sorts of ways to come up with value inside this economy. How do I present my work, or how do I do something so that it has value; so that people want to "buy" it, not necessarily monetarily, but people want to consume it in some way. I don't know. It's a huge problem. Maybe not a problem, but it's certainly something I think a lot of people are thinking about. Then you have net.art, where you have this celebration often—it's quite common—this celebration of this economical collapse; this collapse of the cultural economy. Which is wonderful, you get this euphoric celebration of that, because, you know "okay!" There's this illusion that you're breaking free from economic constraints, but it's not true, you know? You're just manifesting economical systems. But also, then there's another idea, like that's a huge problem too, now. We can't think about anything outside of economic terminology, as dictated by either the capitalist or communist paradigms, which have been really pervasive conceptually. I mean, they both have the same model, they just come at things from different takes. Both those are firmly rooted in "hard" economies, but socially, those ideas have manifested themselves in such a way that everything that are not hard economies are viewed in economical terms, and criticized in economic terms. And I myself, I don't know how to deal with that problem, because I can't help but do it. I don't know; if I try to think about these things, I don't end up anywhere. I mean, I pass through a lot of areas that are perhaps interesting, but it doesn't lead towards any conclusions. Which, is perhaps good or bad, I don't know what kind of value system could be used to even determine that, so... / PROP / Property is such a difficult thing to do, especially when you get into the realm of intellectual property. You know, where are the lines appropriately drawn, if anywhere? Patent law—at least, my understanding of patent law—is that it protects a specific implementation of an idea. In the physical universe, it was easier to determine whether something was infringing on that. There are certain things that I think are perhaps appropriate for patenting. One example might be, Lake Technology has developed a zero latency FFT that uses a windowed... that'll do a convolution in a time domain, and then they'll crossfade quickly to the frequency domain, so that you don't have this inherent latency in a frequency domain based convolution, where you build up the frames to compose one set of data that's long enough to analyze, and then go back again. So they've patented that. That actually is something that, there's a concrete thing, and it's not so common, so that it doesn't seem ridiculous. But patenting things like a scrollbar in a window. Patenting a scrollbar in a window doesn't makes sense for patent law, really, because it's not the implementation of that, it's the idea itself—and there's lots of things presented as though they are the implementation, when they are just ideas in and of themselves, and often times not novel enough. Copyright law doesn't even... There are many different interpretations in copyright law, so if I make a melody, and somebody reuses that melody in different instruments in a different way, you know, it can be a violation of copyright. I mean, I'm not for that stuff either. How can you claim ownership over these ideas? That's where it gets really difficult. It's useful to me, as a musician, or as a developer. It's an economic tool that I certainly take advantage of in certain ways, but if I'm going to think about it not based out of my immediate benefit, then I don't agree with the ideas behind it. / DFNS */ Anyone can go and make the same application, and that's fine. It's not going to deter my development to make my app. There are several things out there that are super similar; I mean, you look at Shake or something like that, this node-based compositing system, or Houdini, which is a node-based animation system, or every editor is now moving towards a nodal view, even if that view is un-editable. There's nothing that we're doing in Jitter that's particularly novel in any way. Or Simulink, MATLAB, which is their block-based program for programming matrices. I don't think that my software development was in any way affected by my understanding of how it was protected by copyright, or not. There's nothing that's really protecting us in any way that would... I mean, the thing that protects us is that there's a lot of work involved in doing it, and you know, somebody else has to do all that work. There are certain things, like copyright of code, where they can't just use it, but if they reprogram something that's essentially, functionally equivalent, I can't sue a person for doing that. There are certain instances where there are lawsuits that are taking place where people attempt to do that, and people are winning in various ways, but it depends. People are really into prediction, and I think it's valuable, perhaps not to gain any insight into where anything may or may not go, but it gives insight into people's understanding of either their own work, or the world at large at the moment. People's projections help to identify the world in which they're living, or their perspective of the world in which they're living. I don't know. At this point in time, I have to make time to get anything done, and I'm not doing that a lot; I'm not finding myself in a position where I choose to work on new music, or new media or whatever, as much as I choose to work on the software. Part of that is probably, from my own position, is that it's maybe much more difficult to deal with these notions of value, and how do I make something that I'm satisfied with and feel fulfilled by, based on some kind of exchange. Whereas software that's just a tool, is perhaps easier to develop and establish whether or not it satisfies certain conditions. Some user wants to be able to invert an image. I mean, the problem is clearly defined, you satisfy that problem, it works, the person buys the product, then it's kind of clear cut. It's easier to evaluate whether or not you've been successful. Not that success is the only thing in life that we should be focusing on... But it's at least an easier problem to not feel trapped and confused and lost within—I mean, as the roaming randomness—or whatever the kind of void that our discussion turned into easier. That's parallel to the same kind of artistic voids you get into when you start questioning these things and evaluating them, and sometimes that's a very difficult space to be in, and I've certainly personally had some moments where I just don't know where to go. I can't make artwork for anyone else, or maybe I can't make artwork at all, or maybe I never have. So to try to identify where I'm going to go with it is almost impossible. INTERVIEW AVAILABLE: JoshKitClayton.int Joshua Kit Clayton is a musician, developer and computer programmer who develops Jitter for Max/MSP. Jitter "extends the Max/MSP programming environment to support realtime manipulation of video, 3D graphics and other data sets... Jitter abstracts all data as multidimensional matrices, so objects that process images can also process audio, volumetric data, 3d vertices, or any numerical information you can get into the computer." (0) As a musician, Joshua Kit Clayton moved towards including realtime video in performances of his musics, musics that can be variously categorized as electronic, laptop [+/or] glitch. recording under the name Kit Clayton, Clayton tours internationally + has released music through Orthlorng Musork (the label he owned + operated) as well as through Cycling '74 (the company for whom Clayton works developing Jitter). Jitter's introduction in 2003 contributed significantly to the expansion of cinematic approaches to realtime audio + video, especially appealing to developers already familiar with Cycling '74's software product Max/MSP. - jonCates 0: data.src: title: Jitter Overview dvr: Cycling '74 uri: http://www.cycling74.com http://criticalartware.net/int/jkC Josh Kit Clayton interviewed by criticalartware coreDevelopers: jonCates bensyverson + Joey Lindsey SAN FRANCISCO CA .US 2003.09.28 interview transcript by: bensyverson http://www.criticalartware.net this criticalartware interview is a shared cultural resource released under a Creative Commons Attribution-NonCommercial-ShareAlike version 2.5 http://creativecommons.org/licenses/by-nc-sa/2.5/