News

October 29, 2009 07:11

Flash Mode Works (I Think)

The dathtml page now has a relaymode option for flash, which sets up a flash applet that will stream your webcam image to the server, and then the server will reserve the image for website viewers. This allows people who are unable to easily configure their network to be able to easily set up a streaming webcam. However, this is not a remarkably fast option. It barely gets more than 1 frame per second, at least with the applet I'm currently using. I'm sure I can either find a better one or write my own eventually, but for the moment, the option to quickly create a new cam is available to those who would otherwise not have it. This also allows one to use the http post option within webcamxp to stream the webcam images, however it will also suffer from the same 1fps limitation.

I have solved some of the memory problems with the camrelay program. By using httprecv for incoming GET requests, the built in timeout features means that more threads are ending sanely and shutting down instead of hanging indefinitely. This has thus resulted in far fewer open threads and fewer allocated image buffers. I've also reduced the stack size of each thread from 10 megs to 256K. I can make that even smaller, but I'm currently using a lot of local variables that are rather large, instead of dynamically allocating them as needed (and perhaps much smaller). In any event, I still have a memory leak in there somewhere, but at least I've fixed the big ones.

Comments(0)

October 12, 2009 08:28

Progress

This weekend I got the camrelay updates done that I've been putting off. I haven't put the new version into production yet, but will likely do so in the next couple of days. This version will feature a 15 second timeout on all connections if no data passes hands when it should be transmitting... and that makes sense. The issue I was having was in regards to quite a few threads hanging somewhere and using up both thread resources as well as about 10 megs of virtual memory each (not sure exactly where the allocation is coming from yet).

Another plus is the relay now works in both directions. It can connect out to a cam server to pull images in, or have cam software with streaming using post connect to it to send the images. I have it working with both the webcamxp post update routine, as well as a flash applet. My efforts so far to test the flash applet have resulted in 2 bluescreens of death.. so far.. so I need to do some extensive testing before I release it for public use. It's possible that the crashing was from splitcam, which is actually a driver... which is able to fuzz around with the system better than an application can. I'm hoping that's all it is. In any event, I'm going to test it out with a real webcam for a while to make sure there aren't any further crashes. Once I deploy this feature, people who have a functioning cam on their system and flash for their browser will be able to have a cam up and running in a matter of seconds, with no software installation required. HOWEVER, I'm going to encourage this as a means of last resort, since it's going to be slow (at least the applet I have now is), and it's going to use up a lot more of my server's bandwidth. Still, if someone is sitting behind a firewall they have no control over (or simply doesn't know how to configure port forwarding), we now have a way to get these people up and running until they're able to figure things out.

That being said, I'm likely going to either eventually code my own flash streamer, modify the code for jpegcam (which is what I'm using), or find something that works slightly better. A fast, smooth, motion jpeg formatted http post command would work nicely as the relay can handle it well. In theory, that should maximise the throughput of any relay connection... in theory. I also discovered yet another tweak I'll have to make to the camrelay. I need to be able to handle chunked transfers. Ugh.. one more tiny little nightmare.

I'm going to need a few things in the near future. First, I need indie Xmas music. Absolutely no restrictions on use would be preferable, but I can go with a "non-commercial only" agreement, so long as it's free and there are no restrictions on broadcasting the audio as part of a video stream. Secondly, I need someone to find the file format for Vixen .vix files. They're xml formated files, but I need to know how to read the sequence. That way I can use vixen to develop a light sequence and use one of my own applications to handle the actual lighting effects. Vixen has a networking option to control the playback from some other network application, but I want my programs to have control of the device. The other option is to run Vixen in a virtualbox and have it use one of the serial based hardware devices. On that system, I'll have a loopbacked serial port so I can have my own application read the Vixen sequence output and reinterpret it in my own application. This way, the lighting display can be controlled by Vixen, but when Vixen isn't in use, lampmaster can handle the lights individual as it does with all the other lights.

Comments(0)

September 27, 2009 05:33

Now for some fun stuff

The months of programming has gotten dull, as one might expect. I REALLY need to concentrate on it some more, but I find myself playing a bit too much half life to distract myself, and I suppose that's really not being very productive. SO.... I've decided to spend a week or so working on some other projects.

First off, I've dug into my collection of cameras and other equipment, mostly just to see what I have. I've barely touched most of it in years, so I kinda needed to do an inventory assessment, if nothing else. I've got 2 large VHS camcorders which seem to work fairly well, one 8mm camcorder that seems to have lighting issues, and a indoor/outdoor day/night camera. All of these use the basic composite jack for video output. I've got several other camcorders that might be useful if I can fix some issues with them, but for now I'll just concentrate on the ones that are functional. I also have a IP Video 9100B video camera server, which takes 4 video inputs, but oddly enough can only actually use one of them at a time... so it's really just a 1 video cam server. It replaces a computer with a capture card and a cam server and does so in a box that's 1" by 3" by 4". It's got a round robin capability, but I'm not sure how to read that yet. More research will be required. I can always ressurect my capture cards for the remaining cameras if need be.

I'm going to start deploying cameras around the house. There has been resistance to this in the past, mostly due to construction related issues, but I really need to get this expanded out of the office and into the hallway, living room, and eventually kitchen. The infrared camera is also an outdoor cam and can be mounted outside. It's also a wide angle camera, so I might mount it so it covers the porch and part of the yard. I'll get a doorbell triggered capture system working again, along with a motion sensor based one. The old house had a window in the door itself, meaning I could mount the camera inside and with a good view of whoever was on the porch. I'll figure something out.

Getting the motion sensors working will require me to get the CM11A functional again. This will make Polk very happy as well, so it's a double bonus. This, however, will require some software work on lampmaster, which I don't plan on doing until I upgrade that server, and THAT won't happen until I strip out all of the nonessential functionality and incorporate it into other servers. So yeah, that one is still going to take a while.

I'm going to give a crack at making a marquee sign again. I long ago found an excellent schematic, which naturally I can't find the paper I printed it on, nor can I find it online anymore. Even so, I might be able to fix the problem I was having way back when and try again to make one. As with the motion sensors, anything device related like this would run through lampmaster, so I'll still need to update that server before a completed project can be deployed. Still, I can always get started.

My ultimate goal for the next few months is to get a Christmas display set up. While I would love to do an outdoor display, the time and logistics, not to mention expense of getting a camera set up outside in a safe, permanant location would not be feasible in the short term, even though there are short term workarounds, such as setting it up in the car, instead of a mailbox. I'm also uncomfortable doing it since the prime display hours would be during the time I'm at work, and I would prefer to be at home when hundreds of strange people are watching the house. I don't want my wife to have to deal with idiots that come up and ring the doorbell thinking that somehow a public display is equivilant to an invitation to come inside.

THEREFORE, I will instead concentrate on an inside display... somewhat smaller, but still a workable demonstration of what I can do outside the next year if it works out well. I've got the software in place to sync lights to music. I'll need to invest some time in putting together the hardware in a safe way (especially being indoors). I'll also need a streaming format that will have video synced to audio so people can observe the whole effect. I have a method using ffmpeg that should be able to serve it using flash, but I still need to spend some time testing this. At least one of the camcorders I mentioned earlier will be deployed for this purpose. Of course, anytime a musical display isn't in operation, all of the individual light componenents will be controllable.

That should keep me busy until Christmas.

Comments(0)

September 17, 2009 23:30

Camserver Finally Done

Ok, that took quite a bit longer than I wanted, but I finally got all of the pages working now with the new camserver. The visitorpage was giving me issues, but it's done now. I do, however, have quite a bit of tweaking to do to make everything work smoothly. There are lots of new fields for both the cam and visitor databases that I need to add to the edit pages, and I still need to make some tweaks to the servers themselves.

The camrelay program is crashproof, but it's still using a LOT of memory. I've got a lingering buffer problem that I have yet to figure out. Here's how it works. The relay program is a webserver I wrote, albiet a very primitive one. It accepts a GET request for an image, and part of the url is the cam number. If the cam number is currently unknown to the relay program, it grabs the url from the database, sets up a thread, and connects and grabs an image and holds it in a buffer. My image buffers are arranged as a linked list, with the first element in the list being the most recently recieved image. Image 0 is always the temporary one. When a request comes in, it grabs the first complete image, which is virutally always image 1, but sometimes image 0. Whenever a new request comes in, it adds a connection id to a linked list of locknodes attached to the individual image buffer, and sends the image, meanwhile setting snarfwaiting to 1 which instructs the snarf thread to go grab another image from the webcam host. As soon as that image is complete, it's available to the client to grab, which triggers another snarf, and this process repeats itself until new requests stop coming in.

The issue at hand here is the locknode list. That list prevents the image from being purged until all connected clients which were receiving the image have received it, or given up. The problem here is that a connection is likely trying to grab that image, or has quit trying, but neglected to release its locknode. That means the image sticks around.. well... forever, even though nobody is ever going to grab it. This makes the image buffer list just continue to grow. I've solved this problem by simply purging any image buffers that are more than a few minutes old. Nobody is sitting around that long waiting on an image.. however, this shouldn't be necessary, and it also points to a possible issue of a connection thread still sitting open, which is going to also consume system resources over time, although not as quickly as images stored in RAM would. Still, its an issue I still need to debug more.

Comments(0)

September 10, 2009 17:23

Well, It's working... mostly

The camserver update is complete, but it was a rougher upgrade than I was hoping for. Not as bad as I feared, but definitely not smooth. There still remain a few lasting challenges, but I've got the majority of the problems solved now. The only remaining problem is the visitors page, but I'm working on that. I've also got the camrelay functioning very well, and I've seen no new bugs in the last several days, but I'm watching. Get a few more cam servers working here and I'll have a better idea if it's fully compatible with all of them, but it should be.

I'll spend a day or two getting everything else fixed and implementing a few new minor features that this latest upgrade has made possible. After that, I'm going to take aim at lampmaster. I'll convert it eventually too, but prior to that, I'm going to strip as much functionality out of it as possible and implement those functions in different servers. Basically, the user updates and dynamic messages will be implemented in different programs. The dynamics will probably best be handled in the messageserver... and I might just stick userupdates in there as well, since that server makes the most sense for the application.

The cam edit page has changed a bit, although not all new functions are available yet. They will be shortly. I also have many more fields to add to the cam page and a few more to add to the visitors page. I also have to update the instruction pages. Anyways, back to work.

Comments(0)

September 02, 2009 20:04

What fun!

Well, I ALMOST made it. I was having some glitchy moments with the harvester, so I didn't get things switched over. Reddit hit first, and then wykop. These events, as desireable as they are, make it really difficult to update the software. Not only am I constantly distracted, but I can't easily debug things when the server is getting slammed. I guess I'm going to need to set myself up a development platform that I can use for testing purposes. It's getting to be too much to manage trying to do the codigin on the production site.

Anyway, I have the harvester fixed... I think. I still need to add in the audio objects database and a means to edit it. And of course, I need to run a recompile on every other program that accesses either the cam or visitor databases. Yes, I know I've said all this before. Just trying to keep everyone updated.

Comments(0)

August 31, 2009 20:11

Staring the Conversion

I've gone as far as I can without pulling the plug on the old system. I'm starting the database conversion process now. What should you expect over the next day?

First off, I'm going to change the .h file so the current camtype becomes oldcamtype and I'll remove the comment tokens from the new camtype to add all the new fields. I'll then code up and convert the old format to the new. The biggest change here is that I'm getting away from the host/port/file method I've been using for ages and I'm just going to have you type in a URL. I'm also not going to have 10 cam options. There will just be one. Screens and alternate views will instead be their own cam record that point back to the main one.

Other new features, in no particular order, will be to store the activate time which can be used for ordering the cams on the page, customized watcher strings, various priority bonuses for cam ordering, alternate account control, so you can allow other specific visitors to modify your cam account. Also will have DMIland data (if I ever get that going again), customized status and caption information for the cam, and some control over relaying.

Audio, screens, and other cam features will now be objects that attach to the cam. This has been bandied about for a while, but it should be reality in the next few days.

Now, for tonight and the rest of this weekend. Once the database has been converted and I'm done making changes to the primary page, I'm going to shut down the server, do a final conversion, recompile and reinstall all the programs, and fire everything back up. The page will be inaccessible for probably a couple hours until I'm positive that everything is working properly. After that, the site will be back up and running as normal.

So here we go!

Comments(0)

August 28, 2009 02:29

Staring the Conversion

I've gone as far as I can without pulling the plug on the old system. I'm starting the database conversion process now. What should you expect over the next day?

First off, I'm going to change the .h file so the current camtype becomes oldcamtype and I'll remove the comment tokens from the new camtype to add all the new fields. I'll then code up and convert the old format to the new. The biggest change here is that I'm getting away from the host/port/file method I've been using for ages and I'm just going to have you type in a URL. I'm also not going to have 10 cam options. There will just be one. Screens and alternate views will instead be their own cam record that point back to the main one.

Other new features, in no particular order, will be to store the activate time which can be used for ordering the cams on the page, customized watcher strings, various priority bonuses for cam ordering, alternate account control, so you can allow other specific visitors to modify your cam account. Also will have DMIland data (if I ever get that going again), customized status and caption information for the cam, and some control over relaying.

Audio, screens, and other cam features will now be objects that attach to the cam. This has been bandied about for a while, but it should be reality in the next few days.

Now, for tonight and the rest of this weekend. Once the database has been converted and I'm done making changes to the primary page, I'm going to shut down the server, do a final conversion, recompile and reinstall all the programs, and fire everything back up. The page will be inaccessible for probably a couple hours until I'm positive that everything is working properly. After that, the site will be back up and running as normal.

So here we go!

Comments(0)

August 23, 2009 03:32

Get ready

Camserver is ready to go.... almost. It's about as ready as I can get it without just unleashing it upon the world. The results, initially, will be somewhat catastrophic. I am absolutely certain, for a short period of time, that the site will break in utterly depressing but amazing ways. Although I'm sure to have the main page functional in a matter of minutes, getting everything working properly will probably take a week or longer. Remember, EVERYTHING that touches the cam or visitor database will be affected by this transition, and it won't be affected well. I'm likely going to be doing some massive replacement of code in almost every program. I've tried over the last few years to make this inevitable transition easier by encapsulating all of the file activity into convienent function calls which could simply be replaced once at the time I switch over. However, many of the programs I have yet to do this with. And I can't really make any more changes to anything else right now because I've already converted many of the function calls over to the new format. I'm going forward with this whether I'm ready or not.

Oh, and just to make it more fun, not only am I going to change how the data is getting retrieved, I'm also adding more fields to the databases themselves. In fact, I'm also eliminating fields from some of them. And I'm adding new databases that point back to the main databases as objects to replace previous functionality that was built in. And no, not all of this code is ready, so some features will actually disappear as soon as the changover takes place. It will be a shortlived inconvienence, but a nasty one none the less. Isn't this exciting!

So here's the process, in a nutshell: I know most of you don't really care, but I'm doing this as much for my own planning as anything, so I'm going to tell you anyway. DEAL WITH IT! Here we go. Before I actually do the changeover, I'm going to make the database conversion, change the init functions to properly zero out the new fields in new records, and recompile all of the programs. I'm past the point of no return at this point, but I haven't yet broken the site. I just can't make any changes anymore without completing the job. It's possible I've already crossed that point, but at least right now I could back out easily if I had to. Nevermind. At this time, I will also do my best to recode everything that needs recoding to work with the new system. I can make sure everything compiles, but I can't really test the programs themselves until I proceed to the next step. So, secondly, all hell breaks loose. I shut down the server, do one last recompile, one more database conversion, and copy over all of the programs to the production directory, restart the webserver and let it rip.

Third, I'll fix all the problems that crop up. I'll just need to be sure I've got enough time to do the second and third steps all in one sitting. That might take hours, so wish me luck with that. Once I'm sure I got all the glitches, I'll start adding in support for the audio and desktops, and music control, and other fun stuff.

After all of that is done I'll move onto the next server. Most likely that will be the live update database. That's currently integrated into lampmaster, but I want to pull that out and put it into some other application. I don't see much point in having it standalone, but I'm not really sure which application to integrate it into. It gets hit often, about once every 10 seconds per user that's on the site at the moment, and is also queried for the admin page and also features like the popup that shows the list of people viewing each cam. I may incorprate this into the cookieserver or possibly the messageserver.

Of course, the big fish is lampmaster itself. Part of this whole conversion process is to pull as much functionality out of lampmaster as possible so when I get to it, it will only handle the devices themselves, and not a slew of other site functions like it currently does. I've already pulled messages out of it, but it still handles the user updates, dynamic functions, and a few other minor things. I'll probably keep waitqueues in there since they relate closely to the device they're associated with. Lampmaster also requires an additional framework feature for slaving. Right now a client can have a persisitant connection to a server and poll the server for updates, however, the server won't automatically send unsolicited messages back to the clients. The other option that currently works are the sync connections. Those connections get sent ALL changes to the databases, as they're intended to keep servers on different machines in sync with each other. However, a third option needs to be added to allow for sending only selected updates or messages to a connected client, but not EVERYTHING. This will be connected as a syncmode just like the sync threads are, but there will be some selective breeding on how messages are sent out. I will likely use the messageserver as a testbed for this, since my synthclient that reads the messages out for me is a good example. While I could just poll the server for updated messages once per second, sending back messages when they appear over a persisitant connection would be cleaner, and a good test for the code.

Also remaining is commentserver. A bunch of new challenges await for that, since I will be applying a lot of new features to comments, including objectifying them to assign them to other objects on the system. After that will be newsserver, imageserver, and probably something that incorporates several of these for use in the DMI documentation project. Then to top it all off, DMICMS itself, and several weeks of testing to get load balancing working perfectly. Ideally I want this all done before October 1. I'm not counting on it though. It'd be nice if I can at least have the core servers finished by then. That will give me enough time to focus on the xmas lighting project. Lots of stuff to do!

Comments(0)

August 19, 2009 20:27

DMI Updates For fun and profit... well, fun anyway

As some of you have noticed, and many of you have inquired about, I am doing a months long software upgrade to the site. I will attempt to answer some of the more frequently asked questions about this process and what it means for the future.

A quick bit of history. The site originally was a simple static page with one cam, one light, and no more than 100 visitors a day. Simple cgi bash scripts that called bottlerocket (a linux based x10 interface program) to turn the lights on and off was all that was needed. As things evolved, I simply made necessary changes to adapt. Even then, until I moved out of the Plano house, things were pretty simple. I had the same number of cams on the site all the time, the same number of lamps, and that kept the dynamic aspect of the site to a bare minimum. Having several hundred visitors on the site at one time was no problem. In fact, the computer I was using then was an order of magnitude less powerful than the one I have hosting the site now, and it could handle 20x the user load. My only constraint at the time was bandwidth.

Over time, however, I added the ability to have people host their own cams, accounts, and numerous other things. This required a lot more computer overhead to build the page. However, as I built this, I did everything as quickly as I could. Therefore, I'm reading flat files for most of the databases. This became problematic as the databases got larger. Long story short, I have a huge hacked-together mess that works... but not very well. I've held off on doing any major publicity for the site, in small part, because I simply can't handle the traffic.

The event that forced me to change my outlook on all of this was the datacenter explosion in June 2008. The site was down for 6 days, and while I do occasional backups of the site in case of something catastrophic, recovering from that would have taken a while if I needed to rebuild everything from backups. I decided at that time I was going to redesign the system so that the databases could be loadbalanced in realtime over multiple servers, at multiple locations. Since this requires recoding practically everything from scratch anyway, it's the perfect opportunity to do it right this time.

All of the databases will be served by several different programs. I'm creating them from scratch instead of using mysql or some other database server for several reasons. First off, a database server is designed to be a jack of all trades approach which means there is more overhead on each command transaction and more storage space is required. With my own programs, I can cheat a bit and save time on various things, as well as be able to support a lot of the workload in the server program itself, things that mysql aren't designed for. I suppose they could be modded, but if I'm going to go to that much work anyway, might as well roll my own.

As for new features, I'm currently just recoding the site to achieve the current functionality. Once I'm finished, new features will be possible. I have a LONG list of stuff I want to do with the site in the next couple of years. I don't expect to be able to accomplish it all, and some things are kinda required as a prerequisite to others, so while a lot of what I'm doing appears dull and boring (even to me), it's necessary none the less for more exciting things. I have determined in the past it's best to not brag endlessly about future projects, especially those which I have no clear roadmap to ever completing. I much prefer to finish something and then unveil it upon a world that is shocked at the discovery. Anyways, I'm off to work, so I'll continue this later.

Comments(0)

More News