My light blogging and lack of workouts indicate that I'm working on server issues. I've posted about ten times like this in the last month. Each time thinking I'd actually solved something. Each time I hadn't.
Why buck the trend? Here we go again...
I just made some big changes to the infrastructure and I'm optimistic that it'll resolve some of the fabled server issues that've been wreaking havoc on my life. T-minus three hours and counting until the inevitable It Didn't Work post.
Last week I dug deep into two areas. 1) The data abstraction layer... which is to say, the way I pull data from the database. 2) The clustering communications that glue all the servers together and make sure that when you change your password on server A it's also updated on server B.
I'm using Hibernate as my data layer. It works well. But one of the symptoms I've seen on the servers when under heavy load is an app server that responds to requests as long as the page doesn't require database access. Lots of patient watching to pattern-recognize this symptom. So I decided that the transaction engine sitting under Hibernate (Atomikos) was causing issues. Spring promised easier transaction management. Bye bye three days of my life. It doesn't.
JGroups ties the cluster together. It's known that when you reach a point between 10 and 15 front-end servers (we're at 4 front-end and 3 back-end) it'll start to create too much overhead. Maybe that effect was happening earlier than expected on my configuration. So I dug deep into config files and protocols and network management. Couldn't find the issue.
After most of last week with no solid leads and lots of hours... like 18+ hour days with only choppy sleep every night... I decided that I needed to start doing more load testing locally to duplicate the issue. With it reproduced locally I'd be able to more quickly try fixes.
Enter JMeter which I've used many times. It simulates users. So I can tell it to act like 100 users hitting dNeero all at once. Or 500. Or 2000.
It took a lot of effort to get all of the conditions on local servers to mimic those in production. And then power at the house started going out every 45 minutes. Only a second at a time but enough to shutdown all of the servers. Annoying.
Finally got the load simulations going. I'd add users and watch the local cluster to see when/how it failed. What I found was that I could run a lot of users over a long period of time and nothing in Hibernate or JGroups bombed. This got me looking back to the Apache mod_proxy and Tomcat thread management.
After lots of testing I came to believe that the way I've been running it in production didn't properly shield individual app servers from load spikes. Apache was just forwarding all requests to the app servers which were blowing up. I had spent all of my time trying to figure out why they blew up but the reality is that all app servers will blow up under enough load. Just take your home computer and start opening applications. Firefox. Outlook. Internet Explorer. Microsoft Word. It'll work up to 30 or so apps. But eventually it'll get too slow and funky stuff'll happen. (No, you can't use this to get out of trouble when you get caught with porn on your screen.)
What I decided to do was look into the load spike scenario. How was the system configured to fail?
Turns out it was configured to crash the app servers, take down the cluster and take three minutes to start back up. Or to lock up and not respond forever. Not cool.
I decided that under very high load I'd rather give the user a message saying "Hey, we're getting hit hard, we'll be back soon." Not giving the user the page they want is a failure but it's a better failure than taking down the whole cluster.
So that's what I did. By limiting the number of connections that mod_proxy will send to each app server. And, very importantly, I told mod_proxy to go ahead and wait for up to 60 seconds to find an app server willing to respond before telling the user we're getting hit hard. This essentially creates a buffer and queue that can hold requests in the case of load spikes. It'll try for a while to respond but if it can't it'll fail (somewhat) gracefully.
It's like a shield for the app servers. And I'm sure it'll fail in a couple hours. But so far it's working as designed. So I'm enjoying the sweet feeling of relief, knowing full well that the next set of server crashes will be even more psychologically devastating.
(Yeah, if you combine optimism and pessimism you get realism.)
Update: Four hours later and nothing but a dramatic improvement. Not perfection... still some issues to hammer out. But I'm more optimistic than not that this puts me on the right path to being able to sleep through the nights again. Tonight will tell the tale, however, as traffic load peaks in the evening. Initial results, good. More results in about 24 hours.
Thanks to Trevelino Keller for having dNeero participate in its Social Status event. TK brought out 40-50 people interested in social media. Five companies got 10 minutes each to describe how they'd help fictional client Zoo Atlanta. And then some Q&A.
I was excited to see so much interest in social media in Atlanta... on a Wednesday morning. Made me think back to meetings ten years ago where nobody knew what a blog was... and I was selling blog software. Not cool. But today people get it and want it.
Paul McKeon from The Content Factor wins the quick blogger award by posting about the event. He also wins the Best Actor award for his enactment of Winston the Warthog and reminded me of our BrightLane radio spot featuring Winston Churchill (second audio link on the post.)
The goal of TK's format was to illustrate their Social Status approach to social media. They evaluate your company's current position in the space and then look to leverage some of the many tools available to make an impact. I appreciated their desire to make the event practical by focusing the participants on a fictional client. This cut through much of the typical "dNeero is awesome" stuff that you get in more traditional formats. It helped us all get more of a conversation going.
Enjoyed it. Look forward to more events like this.
It stands to reason that when I do a long post about how I've achieved something it'll all come down the next day. I decided to extend the scaling to include a third separated table. Woke up at 5:30a and worked on it until about 9 when I launched. It all works. But then it eats too much memory and takes down all of the servers. So I scrambled for a solution but couldn't find one and have reverted to what I had last night. The most cruel part of this is that after weeks of stress over this I woke up feeling great that I had a solution, only to find myself in a miserable day with the site down and ten times more stress. I hate computers.
Update: I've been at this nonstop for almost 14 hours now. I've got it stabilized to the point that it was at this morning. I've tried many things that just haven't worked today. They'll work on testing servers with what I believe to be identical configs and then they'll fail in production (obviously there are differences.) I'm simply exhausted. And I missed Matt and Kelly's wedding. Sorry guys. Happy wedding night.
Athletes, tri peeps and all normal people can skip this blog post. Executive summary: big performance improvement to dNeero releases stress for Mr. Joe Reger who when asked about the launch said "yeah, it's awesome... now where's the next kick to the groin coming from?"
If you've been following the MyThredz that appear at the top of my blog you know that I've been pulling my freaking hair out trying to gain some scalability on the database tier. I've tried a bajillion things (Sequoia/C-JDBC, HA-JDBC, LBPool driver, MySql Cluster.) It's taken a number of weeks and has caused a lot of stress.
Tonight I launched a solution that gets me moving in the right direction. It's not the end-game, but then again it never is. Looking at the system load I found that the key bottleneck in the system is the creation and display of dNeero conversation Flash objects.
Based on the dNeero model where people join conversations and post them to their blogs I certainly knew that there would be a lot of load on that piece of code. Any widget posted to blogs looks a lot like a ddos (distributed denial of service) attack. Lots of computers requesting lots of Flash widgets 24/7. As the user base has grown so too has the load.
No surprise there.
Of course, when I designed the system I had this in mind. My initial approach was to cache all of the Flash objects in memory and then cluster that cache so that if any one server displays a Flash convo on a blog then no other server would have to rebuild it.
In testing this methodology allowed me to serve thousands of objects per second... they were all coming out of memory which is quick.
And this worked for a while. But eventually the amount of memory available to cache Flash objects becomes limited and the cache has to start recycling objects. Even though it uses a LFU (least frequently used) algorithm, when the number of blogs that we serve Flash onto increases over a certain point the hit rate of the cache goes down. In other words, the chances that I've got any specific widget in memory start to decrease dramatically.
When that happens the app servers have to rebuild the Flash widget. Which is expensive. It takes a lot of database calls and a lot of CPU time to rebuild. And each time the servers get restarted they lose the cache and have to rebuild everything.
In short, we outgrew the in-memory cache concept. Lately I've been finding hundreds of threads on each app server in a wait state, waiting for data to build a Flash widget... a widget which in all likelihood has already been build many times before. With the new methodology once it gets built once it's done and persistent in the database cache.
I've been struggling to get the database scaled so that we can utilize multiple servers to handle the load. Note we scaled the app tier a long time ago.
The ideal optimal perfect solution is something like sharding or vertical partitioning or clustering. But I played with those systems for weeks and they're very complex and finicky. They'll work for a little while and then when something goes wrong you spend hours figuring out what's up. This is why when your company grows you hire a fulltime DBA. It really is a fulltime job.
But I can't devote fulltime to the database, not matter how much I like it. I have to be smart. The approach I took was to create a separate database to do nothing but cache the Flash widgets. In this way I can split off one huge piece of load onto its own server. Actually, just making the cache persistent across restarts and large enough to cut down on recycling is a huge step forward.
To implement I created two instances of HibernateUtil... essentially two Hibernate systems running on each app server... one talking to the main database and one talking to the cache database.
The cache database will be large. While the main db is at about 300Mb after a couple years, the cache will probably be around 3400Gb because it holds binary Flash objects in it. Bigger databases mean more risk of bad files, clusters, etc. So I didn't want this binary stuff mixing in with the main data.
If the cache database crashes it's not a huge deal. I'll lose some performance for a while until the app servers can re-generate the most trafficked Flash widgets.
A quick note on that. We've done about 350 conversations. But we store a Flash widget for each user who responds... because they each respond differently. So, we've currently got a little north of 60,000 responses that need corresponding widgets. At about 40k per widget that adds up quickly.
Ok, so the cache db can crash and life goes on.
Currently I've got both the main and the cache db running on the same server. And the performance gain is already evident. The next step is to move the main db to a much faster new server and keep the cache db on the old server. When that happens we'll have a true separation of load on the backend. Which will be nice. And which was the goal.
My next step is to look at a number of other parts of the system to see if I can do table-level partitioning to make performance gains.
It's never really done. Just one step along the way. But it's been driving me nuts. Now I need to get Linux running on the Dell 2850 in my subden and get it to the datacenter.
My original goal was to have a database load solution in place by the time I went to Florida. That means we'd have to get the server down there on Monday for an evening deploy. I don't want to launch new hardware one day before the trip. So if I miss it, I miss it. No big deal. And honestly, I probably will miss it because I won't be able to get Linux configured completely this weekend. I'll give it a shot though. At least the intellectually-challenging portion is taken care of.
A new dNeero client, Delta Community Credit Union. In getting to know about their business I've been surprised that we don't hear more about credit unions given the current market conditions.
If you've got a blog and live in Georgia it'd be a great help to me if you could join the conversation and share your thoughts with DCCU. As always, be dead honest! Thanks!
The servers drive me nuts. Always complaining about something. So I set out late last year to make my life easier. A few minutes ago I launched the project that I call ClusterF. dNeero.com is now running on it and, knock on wood, all seems to be ok.
Things get hairy when your site outgrows its first web server. To scale you start to run it on two servers. This is called clustering. It sounds simple, but it isn't. When a website visitor goes to one of the servers any data they change needs to be visible to website visitors on the other server. So there's a system that synchronizes data between the machines.
Then two servers becomes three, becomes four, etc. Before you know it things are complex and tedious. And, believe me, I design for dead simplicity. I'd rather write 10,000 lines of code than do something manually each time I want to deploy a new build, for example. But complexity creeps in.
My solution was to build my own little Tomcat provisioning system. My design goal was to be able to add a new server to the cluster in two minutes or less. Put a single file onto the server, double click to install, provision the instance and you're up and running. In tests I was able to do two minutes but in production it makes sense to check everything you do so it's realistically more like five minutes to set up a server. Not bad considering that it used to be around an hour... and at that I considered my processes fairly lightweight.
Here's how it works. I install ClusterF onto each server in the cluster. ClusterF includes clean copies of Tomcat and a Java JDK. Using a GUI (set of screens with buttons... a desktop application) I can control any of the servers in the cluster. I can, for example, create a new Tomcat instance.
To create a new Tomcat instance ClusterF copies the clean Tomcat into a deploy directory. It then uses a Java wrapper to make it executable as a service on Windows or Linux... this is done by copying four or five files into the instance's Tomcat directory and editing some config files... all automatically. Next, ClusterF configures Tomcat's server.xml file so that it'll cluster sessions with other instances... or create itself as a separate cluster if so chosen in the GUI.
Once the instances are up and running I can define an App. An App has a set of config properties like database connection strings, etc. I can configure these properties centrally and then ClusterF will make sure they're available as a properties bundle each time an instance is run. I can run multiple Apps per instance by choosing different root directories in the GUI.
Using the GUI I associate Apps to instances, saying essentially that I want App X to run on Servers 1, 2 and 5. I can then deploy a WAR file. ClusterF allows me to choose the WAR file centrally and then it distributes it to the cluster. This was actually quite tricky. I'm clustering using JGroups. I had to break the file into little chunks and send each one as a separate part. JGroups doesn't (yet) handle big payloads well. On the other side I re-assemble the file and process it. Of course, I have to have error checking (MD5 checksum), receipt management, versioning, etc to make sure that the WAR deploys properly.
ClusterF allows me to start/stop all instances running an app with a single button click. And it's cluster smart... meaning that it'll start one instance and give it a 60 second head start so that it'll establish itself as the primary controller in the cluster... then it'll start other instances.
So far I've saved myself time setting up Tomcat instances, managing config files and deploying builds. The last thing I wanted to save time with was restarting app servers.
I built a monitoring system into ClusterF. Each instance pings the others in the cluster periodically to make sure they're up. If they're not, ClusterF can restart them automatically, after set periods of time, etc. I'm essentially trying to make the system self-healing when things like out of memory errors happen.
When things go down I get cell phone pings and emails. I want to be informed, of course, but my hope is that I'll be able to watch ClusterF work through the issue itself from afar. Every now and then I'm sure I'll have to intervene.
ClusterF is a remote control for Tomcat instances. I didn't want Tomcat instances to ever run inside of ClusterF's runtime. Doing so would have introduced another level of possible failure on top of Tomcat. I didn't want that. The Tomcats and ClusterF run independently of each other. If ClusterF goes down I lose some monitoring and self-recovery capability but the Apps stay up. They're decoupled.
Work on ClusterF was always secondary. I got a few hours here and a few hours there for a couple months in Dec 07/Jan 08. Then I got pulled away with other things. A couple weeks ago, with the servers being a pain in the ass, I decided to revive ClusterF. Tonight I finally got it up and running.
Of course, we all know that when I release something it's virtually guaranteed that I'll spend the next 24-48 hours cussing up a storm as it hits the fan. We'll see how this one goes.
Plans for the future of ClusterF? I'll add stuff as I need it. More robust page checking would be nice... the ability to specify a URL and define a string that must come back or else it's defined as a fail. The GUI sucks, but works... I could use some cleaning up there. I'd also like to build a web interface. To do so I'll use NanoHTTPD and write simple pages that have basic status reporting and controls (start, stop, disable, etc.)
The name ClusterF? It's software that makes clustering possible. I can't remember what the F stands for... oh, wait... yes I can. At startup: "Now we're F'd."
Got an article published on TalentZoo.com. Many thanks to Colleen, Web Editor at TalentZoo.com. I'll be doing a column every few weeks... assuming they continue to accept my writings. They've got a bunch of great content for marketers and according to them: "Each installment of your column will be featured for one month on our homepage receiving more than 200,000 impressions and in our email newsletter to over 80,000 recipients." Excellent. I worked dNeero in there where it was relevant but wanted to make sure that I wrote something actually useful to folks. Whether I'm fooling myself and am actually just another buzzword-crazed self-ego-stroking guest columnist remains to be seen. (Not that TZ ever publishes that sort of stuff... but we all know the type.)
Lots of meetings. Met with Wilma Simons and had a great conversation about social media marketing. Met with those potential investors again... and got ourselves a third date! Which is good news, they're great guys and we spent a lot of time learning more about them. Today is an all day working session with The Rainmaker Mark I blogged about last week. We're putting the dNeero presentation into the context of a solutions pitch instead of a product pitch. Interesting and important change of inflection. Sorry for the slow blogging... just busy lately.
One of the potential investors we met with last called me back today. Always a good sign. We'll get together next week to dig into some more detail. He said he and his partner are "very intrigued" by what they saw. Good news is that we have enough powerpoint, business plan and spreadsheet action to fill a writer's strike. But more than anything we just enjoy sharing the dNeero story with others.
rain·mak·er [reyn-mey-ker] -noun Slang. an executive or lawyer with exceptional ability to attract clients, use political connections, increase profits, etc.
We met Mark at the Capital Connections event put on by The Startup Lounge. At the event we talked for about ten or fifteen minutes. Mark is boisterous and animated. Confident. Executive-looking.
This past weekend we saw him again at the SoCon event. He was out looking for opportunities. And we're at the point in our business where we need to generate sales.
So today we met with Mark for about three hours to go over what dNeero has. And what Mark has.
In 1997 our second salesperson hiring mistake told us over dinner at The Food Studio "hey, any salesperson worth their salt can sell himself." Indeed. He was later fired after we caught him lounging around his house with his phone off the hook while he said he was out on sales calls. And he was maybe one of our less horrible salesperson hiring mistakes.
Suffice it to say that we're cautious these days. Mark's not the type that would crumble under an interview, no matter how sinisterly designed. But in a real conversation you can usually get a good sense of what a person's about.
Mark has a very solid understanding of what it takes to sell. He was able to propose ideas and processes on the spot. Able to criticize presentation ideas quickly. You get a sense that he has his own way of working sales. Which is a good thing. Much better than people who just do whatever. Or who sit on their couches.
Which isn't to say he has all the answers. By his own admission he hasn't had any time to dig into our product to understand the value proposition, technical issues, etc. But you get a sense that he'll figure things out.
He looks the part. An adult. Somebody you'd trust. Somebody you might drink a beer with at a NASCAR event. Somebody in a suit. Somebody who's got the pull within his organization to make sure that when the operations guys stumble he can get you a refund... or box seats at a Hawks game.
And, importantly, he's a rainmaker. He knows how to generate sales. While we haven't done a real due diligence on his past (it's not nearly appropriate at this point) we get the sense that through the companies he's created and sold he was the one generating business. It's what he likes to do.
And this is important to Sr. and me because we need a rainmaker. He agreed to go in with us to a meeting we've been working on getting with CocaCola. To see what we do and to help us close the deal. In a sense we'd love for him to take the whole pitch soup to nuts but that's rather too much work after one meeting and a week or two of prep.
After the meeting Sr. and I were cautiously impressed with Mark and his rainmaking skills. We're going to take things slowly. Plenty of time in the universe to make mistakes. Plenty of time to get to know people and get a real sense of what they're about. It's like dating. Because startup businesses are like relationships. You spend a lot of time together. You argue. You want to make things work. From both directions. Mark has to like us... and Sr's kinda prickly and annoying.
We want to find somebody who can take ownership of the sales side of the business. Somebody who can take this product from where it is today and make it an industry standard piece of any social media campaign. The product's poised and ready. The meetings we've had with hundreds of people verify the concept. The users are screaming out for more surveys. The competitors in the market are generating sales with inferior products. Every marketing/pr person we talk to is convinced they can fit it into a campaign in some way.
We're finally in the right place with the right product. While we can figure out the sales piece on our own, in doing so we may let the good timing pass us by. So we want to find a rainmaker. Whether or not Mark will be that person is anybody's guess. In fact, we're not able to answer honestly whether he should be that person. But meeting with him highlights the existence of that skillset and its importance to our business. My role as CEO is to make sure that we have the right resources in place to succeed. My role is not to do all the work myself. Seriously, I need to get some sleep one of these years. Damn servers.
A lot of stuff happening lately. It's amazing what happens when you get away from the computer screen and meet people. Many thanks to Mark for spending the time with us today. We enjoyed it and look forward to the future.
Met with some potential investors who had Alpharetta offices in the same office park where BrightLane was located. Nice guys. Solid experience. They present themselves as not being an angel group. They want to make a bridge loan and then become part of the business, applying their experience and wisdom. Not a bad plan and we'll see how things progress. The ball's in their court to noodle the concept and come back with questions. Or a check. That'd be cool too. Two main pieces of feedback today were about consistent messaging regarding our gross margin calculation and ip protection importance. Good stuff. This meeting was made possible by Dean Trevelino and Genna Keller from Trevelino/Keller and The Startup Council. Thanks guys!
I spent most of today getting the dNeero Reseller Program launched. Most of you are aware of the fact that we're self-funding dNeero surveys and that it's getting more expensive to keep our users happy as more and more people sign up. So we need to get sales. To be sure, we're out there talking to folks.
A couple weeks ago I was saying to myself "jeeze, I'd be willing to give anybody 10% of the sale... I just don't know many people familiar with social networking, blogs, etc." And then I realized that we already have thousands of energetic dNeero users who are clearly motivated by some coin, understand social networks, etc. "Why can't I extend the offer to all dNeero users," I asked myself. Well, I could... but it required some new code. Lately I've been trying to stop development so that I can focus on sales but this development was sales-related. So I went ahead and fit it in between meetings and conferences.
What I'm essentially trying to do is turn thousands of users into thousands of sales people. Not a new model... Amazon generates a big portion of their sales via affiliate programs.
I built it so that it'll also support an internal sales group, once we get one. Every dNeero user now has their own Reseller Code which they can give to researchers and advertisers. If the survey creator uses that code when they create their survey, then they make money. And dNeero has a survey that we're not paying for... which will be a big "Yay" moment.
I set up a Facebook group for resellers. I'll do some promotion of it via email tomorrow. After that I'm thinking about doing some flyers for Tech, GSU, Emory, KSU and any other colleges in town. After that maybe some Craigslist action. And then maybe a posting in some newspapers. It's a real opportunity for those willing to put themselves out there. And it's a huge opportunity for anybody already plugged into the interactive advertising realm.
Tonight was the opening night of SoCon08... the dinner. It was at Maggiano's Cumberland Mall. About 200 social media peeps showed up to confuse, bewilder and amaze each other with buzzwords and technological mumbo jumbo. I did my part and confused no less than twelve people.
Kidding, of course. Most people actually know more than they realize... they just lack the lingo.
I sat at the Marketing/PR table. With seven women and two dudes it was the place to be. We discussed some of the basics and some contemporary topics. More than anything it was nice to get to know some people who were interested in social media.
I met two dNeero users! In the flesh! Michael Mealling of Masten Space Systems is a dNeero user and will send anything you want into space for a rather reasonable fee. I asked him what it would cost to send Bug into space and, on the spot, he calculated that Bug plus the life support systems would probably cost $13,500-ish. Beware Bug... bark at 4:30am one more time and you're outta here. Jacqui Chew of iFusion was a dNeero user before she knew me or Sr. She just stumbled onto it. Pretty cool stuff.
Name dropping time: Kyle Young of multi-taskingwoman.com. Jeff Haynie of appcelerator, who many have said I need to meet... much like they said I needed to meet Allen Graber. Jeffri Epps of foureyes. Mike and Kristin of Radius3. Dunara and Gulbahar, MBA students from the Stans at Kennesaw State University.
So I'm doing something new at this conference. I looked over the list of 250-ish people and picked out about 30 that I'd like to chat with. I printed those out on a sheet along with some summary info grabbed from a proper Googling. Which means that I'm able to focus my discussions more. Basic Conference 101 stuff, I know... but stuff I haven't done in the past. Normally I just walk around and talk to random people.
Speaking of Conference 101... Lance says that you're supposed to take off your name tag when you get your picture taken or when you get up on stage. Never heard that before. And while I'll give it a shot, I realized I had failed tonight when I found myself pumping gas after the event... with my name tag on.
Good work Sherry Heyl and Jeff Haynie... this event is growing and energetic. Just what Atlanta needs.
This afternoon we met with Ann Revell-Pechar of the aptly-named Revell-Pechar Public Relations and Marketing. We met at JavaU in Dunwoody which was a nice relaxed setting. Ann, we learned, spends a lot of time changing the world. Her PR work is focused on entrepreneurial companies who can generate jobs without the potential evils of big business. As such, she has a very international client base. Her perspective made for a feel-good meeting. It's not that any of the other people we talk to don't do good... just that Ann really, really believes in it.
We met Ann a little over a year ago through Lance Weatherby. Ann and her team recommended the term Social Surveys which, as you may have noticed, is dNeero's tagline/subname/whatever-you-call-it. Since our first meeting a lot has changed on our side. It was actually helpful for me to see the progress. We had a user/advertiser chicken and egg problem which we had no idea how to solve. Now we have one side taken care of. We had a scientific rigor problem. Now we have a guy at Rice University ready to do a research paper on dNeero. A lot's happened since we powerpointed last year.
We're trying to find potential clients to use the technology. Ann was kindly willing to think through her clients until she had something of an "aha" moment. From there we talked through the potential client and it sounds like it's a fit. Hopefully we can help Ann deliver on her strategic goals.
Thanks Ann for the time... we're excited to see where this whole thing goes!
Ok, so it's not a real award. But Lance does have some kind words about dNeero on his blog today:
dNeero gets my award for the best promotion, edging the guy holding the light bulb on the pole ala a music festival that I did not get a chance to meet.
Yeah, we totally pwned light bulb guy. And, just as cool, Lance posted his survey answers with a dNeero embed. Great to see and hopefully it'll help Michael Blake create an even better event next time. Thanks Lance!
Just got back from a fun meeting with Stephanie Davis of skirt! magazine. With a solid editorial pedigree including GQ and Self magazines, I knew she'd have some good ideas for dNeero. A quick googling also shows that Stephanie came to Atlanta to jumpstart skirt! magazine by opting out of being one of The Lost Girls who traveled the globe seeking identity, adventure and connection. Such an epic adventure has long been on my list of things to do before I die.
I wasn't sure how to approach the whole church & state issue of editorial & ad sales. Stephanie was clearly on the editorial side but she had a good sense of what was up on the other side of the fence... and the commitment to keep the fence in place. Which is good to see. My sister Kendra and her journalism schooling would appreciate it as well.
Sr., exhibiting his super-human salesperson skills, decided that it'd be good to note and then defend that he voted for McCain while Stephanie voted for somebody else. Sr.'s functionally retarded but Stephanie had a good sense of humor about it and played along. I'm not sure if Stephanie wants her vote public but suffice it to say she didn't vote for McCain.
Stephanie was kind enough to introduce us to others at Cox, their parent company. They have one internal marketing organization that runs campaigns for skirt!, Mundo and others. We're hoping that there's some way we can help them leverage the social media space. While I don't want to put words into Stephanie's mouth, it would appear on the surface that the dNeero concept is intriguing to her and may have some applicability. Ok, so I put some words in there. She can edit me out if she likes. Or destroy me with a Tall Skinny Blogger expose of some sort. Which really wouldn't be cool because the ladies are totally my key target demo. Lol.
Thanks to Stephanie for the time today. We enjoyed it. Now why do I feel like this blog post is gonna come back to me marked up with a red pen?
The task was fairly straight-forward. Go to the recent AiMA event and get contact info for three pre-defined players in the Atlanta internet/interactive community. Heather and I had already planned a date night so I wouldn't be able to attend. So Sr. went alone. And he came back with not three but five excellent contacts that're excited to sit down with us and get a demo of dNeero. One thing we did well this time was pre-target some people who made sense to get in front of. Usually we just wander around those social events and hope to find somebody. Having a list of names ahead of time seems to help. AiMA brought out some high level executive types for a panel discussion. Sr. gave each of them a quick intro to dNeero and secured some interest for a demo later on. Five points to Sr. Good work big guy!
Just launched a redesigned dNeero homepage. More compact near the top. A little poor man's Flash... image rollovers. Moving away from the term Blogger to the term Social People because many users are from Facebook and other social networks. But the big change is in the dynamic content. Now that we've got a lot of activity we're able to list things that users do. Attaching two pics... one of the old page and one of the new. The new one is the tall one.
Update 4:26AM: As is usually the case there were a few bugs to work out. Made some changes and redeployed.
Up late launching some new code. Here are the details. Now I need to calm down a bit... launching new stuff is always a bit of a rush. Getting a new feature ready for business takes energy, enthusiasm, creativity and alertness. By the time I launch I'm excited about other people seeing the new stuff. And I'm also terrified that I've hosed some part of the system and will be up all night attempting to remedy the situation. It usually takes 30 min to an hour of watching the back-end server logs to get a warm fuzzy feeling that things are working well. Then I can grab some sleep.
Just noticed that dNeero has hit a million survey displays. This means that across all the people who are blogging dNeero surveys we've displayed a survey inside of a blog (or social network) one million times. Not a very huge number at all in the grand scale of CPM business models but that side of things is only a portion of our proposition. The seven digit million mark is a shiny, crisp number. Baby steps. Thanks to all the bloggers who've trusted us to take part in their online life!
SufferFest 08 Day 4
